(integ-tests-alpha,custom-resources): snapshots are always outdated

kornicameister commented 1 week ago

Describe the bug

After upgrading to 2.146.0 from 2.145.0 where https://github.com/aws/aws-cdk/issues/30418 was introduced I am observing that running integration tests via integ-runner always fails. Snapshot is being reported as outdated, like on screenshot above. Although I can successfully rebuild/rerun tests and I can observe a change in snapshots next time I run tests I once again observe situation from screenshot

Once I had downgraded to 2.145.0 on both aws-cdk-lib, @aws-cdk/integ-runner and @aws-cdk/integ-tests-alpha to 2.145.0 tests are passing without a problem.

Behavior suggests a regression of sort around: https://github.com/aws/aws-cdk/issues/30418 Also the problem occurs, I believe, around custom resources I am using.

Expected Behavior

Screenshot are deterministically stored and next checks aren't failing.

Current Behavior

Reproduction Steps

N/A

Possible Solution

N/A

Additional Information/Context

N/A

CDK CLI Version

2.146.0

Framework Version

No response

Node.js Version

18.16.0

OS

MacOS sierra

Language

TypeScript

Language Version

5.4.5

Other information

No response

kornicameister commented 1 week ago

Hmm.. I thought I had fixed by downgrading to 2.145.0 but now I am running a test that has changed custom resource and it fails with same reason. This time going down to 2.144.0 fixes the problem consistently.

pahud commented 3 days ago

Yes I think https://github.com/aws/aws-cdk/pull/30418 would make your snapshot need to update as it makes logApiResponseData: true to logApiResponseData: undefined. This is a bug fix and existing snapshots would need to update.

kornicameister commented 3 days ago

@pahud but they are always updating and never actually doing that.

aws / aws-cdk