Closed youngbupark closed 1 year ago
The error is expected. Looks like it was because of network blip in 1ES VM network.
The following tests are failing:
=== NAME Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
cli.go:379: [rad] Building /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep...
cli.go:379: [rad] Deploying template '/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep' into environment 'kind-radius' from workspace 'kind-radius'...
cli.go:379: [rad]
cli.go:379: [rad] Deployment In Progress...
cli.go:379: [rad]
cli.go:379: [rad] Error: {
cli.go:379: [rad] "code": "DeploymentFailed",
cli.go:379: [rad] "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
cli.go:379: [rad] "details": [
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "ResourceDeploymentFailure",
cli.go:379: [rad] "message": "An exception was encountered. Current deployment operation has failed. Error message: 'An error occurred while sending the request.'. Exception: 'System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\n ---\u003e System.Net.Sockets.SocketException (104): Connection reset by peer\n --- End of inner exception stack trace ---\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource\u003cSystem.Int32\u003e.GetResult(Int16 token)\n at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)\n at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)\n at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\n at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)'"
cli.go:379: [rad] }
cli.go:379: [rad] ]
cli.go:379: [rad] }
cli.go:379: [rad]
cli.go:379: [rad] TraceId: aa091f379c6abfd211d76fea71145b3c
cli.go:379: [rad]
deployexecutor.go:66:
Error Trace: /usr/local/vss-agent/2.304.0/_work/radius/radius/test/step/deployexecutor.go:66
/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/corerptest.go:259
Error: Received unexpected error:
code DeploymentFailed: err At least one resource deployment operation failed. Please see the details for the specific operation that failed.
Test: Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
Messages: failed to deploy deploy testdata/corerp-resources-container-workload.bicep
...
=== NAME Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
cli.go:379: [rad] Building /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep...
cli.go:379: [rad] Deploying template '/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep' into environment 'kind-radius' from workspace 'kind-radius'...
cli.go:379: [rad]
cli.go:379: [rad] Deployment In Progress...
cli.go:379: [rad]
cli.go:379: [rad] Error: {
cli.go:379: [rad] "code": "DeploymentFailed",
cli.go:379: [rad] "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
cli.go:379: [rad] "details": [
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "Conflict",
cli.go:379: [rad] "message": "The target resource is in progress state: Updating."
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] }
cli.go:379: [rad] ]
cli.go:379: [rad] }
cli.go:379: [rad]
cli.go:379: [rad] TraceId: f02ef9f4a14ab508e9d5e1aecf867ff2
cli.go:379: [rad]
deployexecutor.go:66:
Error Trace: /usr/local/vss-agent/2.304.0/_work/radius/radius/test/step/deployexecutor.go:66
/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/corerptest.go:259
Error: Received unexpected error:
code DeploymentFailed: err At least one resource deployment operation failed. Please see the details for the specific operation that failed.
Test: Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
Messages: failed to deploy deploy testdata/corerp-resources-container-workload.bicep
...
--- FAIL: Test_MongoDB_Recipe_ContextParameter (210.85s)
--- FAIL: Test_MongoDB_Recipe_ContextParameter/deploy_testdata/corerp-resources-mongodb-recipe-context.bicep (149.40s)
=== NAME Test_MongoDB_Recipe_ContextParameter/deploy_testdata/corerp-resources-mongodb-recipe-context.bicep
cli.go:379: [rad] Building /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-mongodb-recipe-context.bicep...
cli.go:379: [rad] Deploying template '/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-mongodb-recipe-context.bicep' into environment 'kind-radius' from workspace 'kind-radius'...
cli.go:379: [rad]
cli.go:379: [rad] Deployment In Progress...
cli.go:379: [rad]
cli.go:379: [rad] Error: {
cli.go:379: [rad] "code": "DeploymentFailed",
cli.go:379: [rad] "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
cli.go:379: [rad] "details": [
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "OK",
cli.go:379: [rad] "message": ""
cli.go:379: [rad] },
cli.go:379: [rad] {
cli.go:379: [rad] "code": "ResourceDeploymentFailure",
cli.go:379: [rad] "message": "Failed",
cli.go:379: [rad] "details": [
cli.go:379: [rad] {
cli.go:379: [rad] "code": "Internal",
cli.go:379: [rad] "message": "GET [https://10.96.0.1:443/apis/api.ucp.dev/v1alpha3/planes/deployments/local/resourcegroups/kind-radius/providers/Microsoft.Resources/deployments/recipe1685004313867393973/operationStatuses/08585166025715004979\n--------------------------------------------------------------------------------\nRESPONSE](https://10.96.0.1/apis/api.ucp.dev/v1alpha3/planes/deployments/local/resourcegroups/kind-radius/providers/Microsoft.Resources/deployments/recipe1685004313867393973/operationStatuses/08585166025715004979/n--------------------------------------------------------------------------------/nRESPONSE) 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"id\": null,\n \"resourceId\": null,\n \"name\": null,\n \"status\": \"Failed\",\n \"startTime\": null,\n \"endTime\": null,\n \"percentComplete\": null,\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"target\": null,\n \"message\": \"At least one resource deployment operation failed. Please see the details for the specific operation that failed.\",\n \"details\": [\n {\n \"code\": \"ResourceDeploymentFailure\",\n \"target\": null,\n \"message\": \"An exception was encountered. Current deployment operation has failed. Error message: 'An error occurred while sending the request.'. Exception: 'System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---\u003e System.Net.Sockets.SocketException (104): Connection reset by peer\\n --- End of inner exception stack trace ---\\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource\u003cSystem.Int32\u003e.GetResult(Int16 token)\\n at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)\\n at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)\\n at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)'\",\n \"details\": null,\n \"additionalInfo\": null\n }\n ],\n \"additionalInfo\": null\n },\n \"properties\": null,\n \"operations\": null\n}\n--------------------------------------------------------------------------------\n"
cli.go:379: [rad] }
cli.go:379: [rad] ]
cli.go:379: [rad] }
cli.go:379: [rad] ]
cli.go:379: [rad] }
cli.go:379: [rad]
cli.go:379: [rad] TraceId: 65fe97eb58b5f8f386acf6f3bd1774c4
cli.go:379: [rad]
deployexecutor.go:66:
Error Trace: /usr/local/vss-agent/2.304.0/_work/radius/radius/test/step/deployexecutor.go:66
/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/corerptest.go:259
Error: Received unexpected error:
code DeploymentFailed: err At least one resource deployment operation failed. Please see the details for the specific operation that failed.
Test: Test_MongoDB_Recipe_ContextParameter/deploy_testdata/corerp-resources-mongodb-recipe-context.bicep
Messages: failed to deploy deploy testdata/corerp-resources-mongodb-recipe-context.bicep
After the investigation, the behavior is very similar to SNAT port exhaustion. We already provisioned 2 PIPs to LB, so I didn't think our tests consume all ports. I increased to 3 Public IPs for outgoing traffic. Let's see how it goes cc/ @rynowak @shalabhms @sylvainsf @vinayada1
Bug information
Test_AWSRedeployWithUpdatedResourceUpdatesResource is constantly failing.
Test run link: https://github.com/project-radius/radius/actions/runs/5072749040/jobs/9112387126
AB#7969