search

radius-project / radius

Radius is a cloud-native, portable application platform that makes app development easier for teams building cloud-native apps.
https://radapp.io
Apache License 2.0
1.45k stars 92 forks source link

Functional test failing #5597

Closed youngbupark closed 1 year ago

youngbupark commented 1 year ago

Bug information

Test_AWSRedeployWithUpdatedResourceUpdatesResource is constantly failing.

Test run link: https://github.com/project-radius/radius/actions/runs/5072749040/jobs/9112387126

=== NAME  Test_AWSRedeployWithUpdatedResourceUpdatesResource
    corerptest.go:317: finished deleting radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e
    corerptest.go:322: validating deletion of AWS resource for radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e (attempt 1/5)
    corerptest.go:326: checking existence of resource radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e failed with err: operation error CloudControl: GetResource, https response error StatusCode: 400, RequestID: 71dc0393-eea3-47e1-a89a-7e24cc8111f8, ResourceNotFoundException: AWS::S3::Bucket Handler returned status FAILED: Bucket not found (HandlerErrorCode: NotFound, RequestToken: 9a25c591-6b9e-49a7-ae7b-cf7a767dbc97)
    corerptest.go:336: finished validation of deletion of AWS resource radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e for radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e
    corerptest.go:313: deleting radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e
    corerptest.go:317: finished deleting radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e
    corerptest.go:322: validating deletion of AWS resource for radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e (attempt 1/5)
    corerptest.go:326: checking existence of resource radiusfunctionaltestbucket-f4c419ba-f242-4aa7-b6cd-a4ca9dc2939e failed with err: operation error CloudControl: GetResource, https response error StatusCode: 400, RequestID: 0406c468-1ebe-4ea7-a6c8-a1ab2e2a6a74, ResourceNotFoundException: AWS::S3::Bucket Handler returned status FAILED: Bucket not found (HandlerErrorCode: NotFound, RequestToken: 7523387d-dba0-4b43-b40e-2e3fabf4f616)
image

AB#7969

youngbupark commented 1 year ago

The error is expected. Looks like it was because of network blip in 1ES VM network.

image
youngbupark commented 1 year ago

The following tests are failing:

=== NAME  Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
    cli.go:379: [rad] Building /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep...
    cli.go:379: [rad] Deploying template '/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep' into environment 'kind-radius' from workspace 'kind-radius'...
    cli.go:379: [rad] 
    cli.go:379: [rad] Deployment In Progress...
    cli.go:379: [rad] 
    cli.go:379: [rad] Error: {
    cli.go:379: [rad]   "code": "DeploymentFailed",
    cli.go:379: [rad]   "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
    cli.go:379: [rad]   "details": [
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "ResourceDeploymentFailure",
    cli.go:379: [rad]       "message": "An exception was encountered. Current deployment operation has failed. Error message: 'An error occurred while sending the request.'. Exception: 'System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\n ---\u003e System.Net.Sockets.SocketException (104): Connection reset by peer\n   --- End of inner exception stack trace ---\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource\u003cSystem.Int32\u003e.GetResult(Int16 token)\n   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)\n   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)'"
    cli.go:379: [rad]     }
    cli.go:379: [rad]   ]
    cli.go:379: [rad] }
    cli.go:379: [rad] 
    cli.go:379: [rad] TraceId:  aa091f379c6abfd211d76fea71145b3c
    cli.go:379: [rad] 
    deployexecutor.go:66: 
            Error Trace:    /usr/local/vss-agent/2.304.0/_work/radius/radius/test/step/deployexecutor.go:66
                                        /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/corerptest.go:259
            Error:          Received unexpected error:
                            code DeploymentFailed: err At least one resource deployment operation failed. Please see the details for the specific operation that failed.
            Test:           Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
            Messages:       failed to deploy deploy testdata/corerp-resources-container-workload.bicep
...
=== NAME  Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
    cli.go:379: [rad] Building /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep...
    cli.go:379: [rad] Deploying template '/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-container-workload.bicep' into environment 'kind-radius' from workspace 'kind-radius'...
    cli.go:379: [rad] 
    cli.go:379: [rad] Deployment In Progress...
    cli.go:379: [rad] 
    cli.go:379: [rad] Error: {
    cli.go:379: [rad]   "code": "DeploymentFailed",
    cli.go:379: [rad]   "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
    cli.go:379: [rad]   "details": [
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "Conflict",
    cli.go:379: [rad]       "message": "The target resource is in progress state: Updating."
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     }
    cli.go:379: [rad]   ]
    cli.go:379: [rad] }
    cli.go:379: [rad] 
    cli.go:379: [rad] TraceId:  f02ef9f4a14ab508e9d5e1aecf867ff2
    cli.go:379: [rad] 
    deployexecutor.go:66: 
            Error Trace:    /usr/local/vss-agent/2.304.0/_work/radius/radius/test/step/deployexecutor.go:66
                                        /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/corerptest.go:259
            Error:          Received unexpected error:
                            code DeploymentFailed: err At least one resource deployment operation failed. Please see the details for the specific operation that failed.
            Test:           Test_Storage/deploy_testdata/corerp-resources-container-workload.bicep
            Messages:       failed to deploy deploy testdata/corerp-resources-container-workload.bicep
...
--- FAIL: Test_MongoDB_Recipe_ContextParameter (210.85s)
    --- FAIL: Test_MongoDB_Recipe_ContextParameter/deploy_testdata/corerp-resources-mongodb-recipe-context.bicep (149.40s)
=== NAME  Test_MongoDB_Recipe_ContextParameter/deploy_testdata/corerp-resources-mongodb-recipe-context.bicep
    cli.go:379: [rad] Building /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-mongodb-recipe-context.bicep...
    cli.go:379: [rad] Deploying template '/usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/resources/testdata/corerp-resources-mongodb-recipe-context.bicep' into environment 'kind-radius' from workspace 'kind-radius'...
    cli.go:379: [rad] 
    cli.go:379: [rad] Deployment In Progress...
    cli.go:379: [rad] 
    cli.go:379: [rad] Error: {
    cli.go:379: [rad]   "code": "DeploymentFailed",
    cli.go:379: [rad]   "message": "At least one resource deployment operation failed. Please see the details for the specific operation that failed.",
    cli.go:379: [rad]   "details": [
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "OK",
    cli.go:379: [rad]       "message": ""
    cli.go:379: [rad]     },
    cli.go:379: [rad]     {
    cli.go:379: [rad]       "code": "ResourceDeploymentFailure",
    cli.go:379: [rad]       "message": "Failed",
    cli.go:379: [rad]       "details": [
    cli.go:379: [rad]         {
    cli.go:379: [rad]           "code": "Internal",
    cli.go:379: [rad]           "message": "GET [https://10.96.0.1:443/apis/api.ucp.dev/v1alpha3/planes/deployments/local/resourcegroups/kind-radius/providers/Microsoft.Resources/deployments/recipe1685004313867393973/operationStatuses/08585166025715004979\n--------------------------------------------------------------------------------\nRESPONSE](https://10.96.0.1/apis/api.ucp.dev/v1alpha3/planes/deployments/local/resourcegroups/kind-radius/providers/Microsoft.Resources/deployments/recipe1685004313867393973/operationStatuses/08585166025715004979/n--------------------------------------------------------------------------------/nRESPONSE) 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": null,\n  \"resourceId\": null,\n  \"name\": null,\n  \"status\": \"Failed\",\n  \"startTime\": null,\n  \"endTime\": null,\n  \"percentComplete\": null,\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"target\": null,\n    \"message\": \"At least one resource deployment operation failed. Please see the details for the specific operation that failed.\",\n    \"details\": [\n      {\n        \"code\": \"ResourceDeploymentFailure\",\n        \"target\": null,\n        \"message\": \"An exception was encountered. Current deployment operation has failed. Error message: 'An error occurred while sending the request.'. Exception: 'System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.\\n ---\u003e System.Net.Sockets.SocketException (104): Connection reset by peer\\n   --- End of inner exception stack trace ---\\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource\u003cSystem.Int32\u003e.GetResult(Int16 token)\\n   at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)\\n   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)\\n   at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)\\n   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)'\",\n        \"details\": null,\n        \"additionalInfo\": null\n      }\n    ],\n    \"additionalInfo\": null\n  },\n  \"properties\": null,\n  \"operations\": null\n}\n--------------------------------------------------------------------------------\n"
    cli.go:379: [rad]         }
    cli.go:379: [rad]       ]
    cli.go:379: [rad]     }
    cli.go:379: [rad]   ]
    cli.go:379: [rad] }
    cli.go:379: [rad] 
    cli.go:379: [rad] TraceId:  65fe97eb58b5f8f386acf6f3bd1774c4
    cli.go:379: [rad] 
    deployexecutor.go:66: 
            Error Trace:    /usr/local/vss-agent/2.304.0/_work/radius/radius/test/step/deployexecutor.go:66
                                        /usr/local/vss-agent/2.304.0/_work/radius/radius/test/functional/corerp/corerptest.go:259
            Error:          Received unexpected error:
                            code DeploymentFailed: err At least one resource deployment operation failed. Please see the details for the specific operation that failed.
            Test:           Test_MongoDB_Recipe_ContextParameter/deploy_testdata/corerp-resources-mongodb-recipe-context.bicep
            Messages:       failed to deploy deploy testdata/corerp-resources-mongodb-recipe-context.bicep
youngbupark commented 1 year ago

After the investigation, the behavior is very similar to SNAT port exhaustion. We already provisioned 2 PIPs to LB, so I didn't think our tests consume all ports. I increased to 3 Public IPs for outgoing traffic. Let's see how it goes cc/ @rynowak @shalabhms @sylvainsf @vinayada1