aws / aws-toolkit-vscode

Amazon Q, CodeCatalyst, Local Lambda debug, SAM/CFN syntax, ECS Terminal, AWS resources
https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.amazon-q-vscode
Apache License 2.0
1.41k stars 346 forks source link

fix: unreliable test "submitFeedbackListener" #5232

Closed justinmk3 closed 1 week ago

justinmk3 commented 1 week ago

Problem:

unreliable test:

submitFeedbackListener
  submits feedback for Amazon Q, disposes, and handles errors:

 AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
 assert.ok(postStub.calledOnceWithExactly({ comment: comment, sentiment: sentiment }))
 + expected - actual
 -false
 +true
 at Context.<anonymous> (src/test/feedback/commands/submitFeedbackListener.test.ts:33:20)

This is because of a race: CodeTransform sets a global "jobId", which may be used by the feedback form, which causes unexpected results in the feedback tests.

Solution:

Remove random CodeTransform special-case from FeedbackWebview.

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

justinmk3 commented 1 week ago

Looks like there is a race with https://github.com/aws/aws-toolkit-vscode/blob/a124aa6f5349fa2421df4efe32473881a1e93613/packages/core/src/test/codewhisperer/commands/transformByQ.test.ts#L83

  2 failing
  1) submitFeedbackListener
       submits Amazon Q feedback, disposes, and shows message on success:

      AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

  {
+   comment: 'comment\n\nQ CodeTransform jobId: abc-123',
-   comment: 'comment',
    sentiment: 'Positive'
  }
      + expected - actual

       {
      -  "comment": "comment\n\nQ CodeTransform jobId: abc-123"
      +  "comment": "comment"
         "sentiment": "Positive"
       }

      at Context.<anonymous> (src/test/feedback/commands/submitFeedbackListener.test.ts:40:20)

  2) submitFeedbackListener
       submits AWS Toolkit feedback, disposes, and shows message on success:

      AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

  {
+   comment: 'comment\n' +
+     '\n' +
+     'Q CodeTransform jobId: abc-123\n' +
+     '\n' +
+     'Q CodeTransform jobId: abc-123\n' +
+     '\n' +
+     'Q CodeTransform jobId: abc-123',
-   comment: 'comment',
    sentiment: 'Positive'
  }
      + expected - actual

       {
      -  "comment": "comment\n\nQ CodeTransform jobId: abc-123\n\nQ CodeTransform jobId: abc-123\n\nQ CodeTransform jobId: abc-123"
      +  "comment": "comment"
         "sentiment": "Positive"
       }

      at Context.<anonymous> (src/test/feedback/commands/submitFeedbackListener.test.ts:40:20)