integ-tests-alpha: cdk tests don't run if we update assertion

phch commented 1 year ago

Describe the bug

    "@aws-cdk/integ-runner": "^2.73.0-alpha.0",
    "@aws-cdk/integ-tests-alpha": "^2.73.0-alpha.0",

Command: integ-runner --directory ./integ-tests --parallel-regions us-east-1 --update-on-failed

We created integration tests for our CDK application and there was a snapshot taken on a successful test pass. When I went in and changed some parameters of the integ-tests (changed the assertion expected value from a Dynamo GetItem call) it failed to run the test again and incorrectly passed each subsequent run of the command above. I ran npm run build before running the integ-runner.

I had add the --force flag to the integ runner command to get things to run. Filing a bug because this is unexpected behavior and allows for test cases that are incorrect to be passed.

    integ.assertions
      .awsApiCall('DynamoDB', 'getItem', {
        TableName: stackUnderTest.tableName,
        Key: { id: { S: id } },
      })
      /**
       * Expect the enriched message to be returned.
       */
      .expect(
        ExpectedResult.objectLike({
          Item: {
            id: {
              S: id,
            },
            message: {
              S: message,
            },
            additionalAttr: {
              S: 'enriched', // Changing this line does not make test run again
            },
          },
        }),
      )

Expected Behavior

Tests should be rerun if the integration tests definition change.

Current Behavior

After changing the assertion in the integration tests definition the tests still pass correctly (only compares snapshot). No changes were made to application definition.

> integ-runner --directory ./integ-tests --parallel-regions us-east-1

Verifying integration test snapshots...

  UNCHANGED  integ.sns-sqs-ddb 5.384s

Snapshot Results: 

Tests:    1 passed, 1 total

Reproduction Steps

Clone repo: https://github.com/aws-samples/cdk-integ-tests-sample
Run npm i && npm build && npm run integ-test - should run fine.
Update integration test assertion and run npm run build && npm run integ-test - passes successfully even though assertion changed.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.75.1 (build aa30a30)

Framework Version

No response

Node.js Version

8.6.0

OS

Mac

Language

Typescript

Language Version

Typescript (4.6.3)

Other information

No response

peterwoodworth commented 1 year ago

I think it makes sense that you would expect them to run if they have been updated as you've described. However, I'm not sure if that is intentional or not. At the least, we should specify in documentation exactly when the integration tests will fully run or not

guiyom-e commented 1 year ago

This issue is similar to this one: https://github.com/aws/aws-cdk/issues/24215, where some existing tests pass thanks to snapshots, but are deprecated and fail when --force is used to bypass snapshots.

tripl3a commented 17 hours ago

I'm having the same issue. Neither changes to the test file nor the file under test are detected/considered unless --force is used. Am using integ-runner version 2.148.0-alpha.0.

aws / aws-cdk