CI: failing CodeCatalyst E2E test: updateDevEnvironment causes InternalException

[justinmk3]

Problem

CC e2e tests failing on https://github.com/aws/aws-toolkit-vscode/commit/61bbc248221ac21e30fa74aaec20a12cae59c10a

  1) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         connects to a running Dev Environment:
     Error: Dev Environment failed to start (FAILED): test-alias-16817597…
      at /codebuild/output/src753672424/src/github.com/aws/aws-toolkit-vscode/src/shared/clients/codecatalystClient.ts:534:27
      at processTicksAndRejections (node:internal/process/task_queues:96:5)

  2) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         stops a running Dev Environment, then starts it up again:
     Error: Dev env 8c7c9a6c-4ed4-4b5c-8626-33024bc7b708 did not transition to one of the following statuses before timeout: "RUNNING"
      at backend_1 (src/testE2E/codecatalyst/client.test.ts:528:23)
      at Generator.next (<anonymous>)
      at fulfilled (src/testE2E/codecatalyst/client.test.ts:9:58)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)

  3) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         updates the properties of an existing dev environment:
     InternalException: An internal error has occurred while processing your request

Full build log

• buildlog.txt.gz

Solution

• ✅ does it need to be reported to the service team?
• is it a false positive? can the test harness make this clearer (by reporting relevant info, but not too much info, in the logs), or guard against it?
• print the related request-id with the failure?

[JadenSimon]

InternalException: An internal error has occurred while processing your request

This is possibly a backend regression with updateDevEnvironment

[justinmk3]

Also noticed that during successful CC e2e test run, logs show AccessDeniedException for GetDevEnvironment and DeleteDevEnvironment. Looks like this is expected because it's cleanup code in a before/after block. Ideally, we would not log expected errors in the build log.

2023-04-17 15:36:17 [ERROR]: API request failed: getDevEnvironment
params: {
  spaceName: 'aws-toolkits-test-space',
  projectName: 'aws-toolkit-integ-test-project-isolated',
  id: '801f2b15-10d9-41e8-a1c9-a9ff19846dac'
}
error: {
  message: "You don't have permission to perform this operation: GetDevEnvironment",
  code: 'AccessDeniedException',
  time: 2023-04-17T15:36:17.136Z,
  statusCode: 403,
  retryable: false,
  retryDelay: 33.69996626932179
}
headers: {
  'x-amz-apigw-id': undefined,
  'x-amz-cf-id': '-MTWdn4TIlcKMr_U2nM9DSMZG8JVCsIeio7VBzwoXUqXvhp8e5H-JA==',
  'x-amz-cf-pop': 'HIO52-P1',
  'x-amzn-remapped-content-length': undefined,
  'x-amzn-remapped-x-amzn-requestid': undefined,
  'x-amzn-requestid': '51dba5dc-f783-4d63-86d4-95e9b53ac1c0',
  'x-amzn-served-from': 'us-west-2',
  'x-amzn-trace-id': undefined,
  'x-cache': 'Error from cloudfront',
  'x-request-id': undefined
}

[justinmk3]

Also noticed that during successful CC e2e test run, logs show AccessDeniedException for GetDevEnvironment and DeleteDevEnvironment. Looks like this is expected because it's cleanup code in a before/after block. Ideally, we would not log expected errors in the build log.

https://github.com/aws/aws-toolkit-vscode/pull/3357

[justinmk3]

All E2E jobs are now failing because of throttling:

  1) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         creates multiple dev envs, lists them, then deletes them:
     ThrottlingException: The request was denied due to request throttling.

  2) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         "after all" hook for "lists all projects for the given user":
     ThrottlingException: The request was denied due to request throttling.

[justinmk3]

print the related request-id with the failure?

Not needed since the dev env id is printed and that's enough for troubleshooting.

e2e tests were reenabled in https://github.com/aws/aws-toolkit-vscode/pull/3827 and are passing now.

[justinmk3]

Getting throttled again on mainline. https://github.com/aws/aws-toolkit-vscode/commit/c1005e9151b74878278eec4772e3cbf307f6fe89 https://github.com/aws/aws-toolkit-vscode/commit/c55209191779aefde8f23b0269e1f98c7270c0d1

  11 passing (3m)
  3 failing
  1) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         creates a Dev Environment with a different instance type + storage:
     ThrottlingException: The request was denied due to request throttling.

  2) Test how this codebase uses the CodeCatalyst API
       Dev Environment functionality
         creates multiple dev envs, lists them, then deletes them:
     ThrottlingException: The request was denied due to request throttling.