aws-cdk-lib/assertions: Cannot run testing against ProductStack objects

[pahud]

Reopening this issue for https://github.com/aws/aws-cdk/issues/24988 from thebillest

Discussed in https://github.com/aws/aws-cdk/discussions/24989

^{Originally posted by **thebillest** April 7, 2023} ### Describe the bug When trying to perform testing with cdk-nag, according to the methodology listed [on this AWS blog](https://aws.amazon.com/blogs/devops/manage-application-security-and-compliance-with-the-aws-cloud-development-kit-and-cdk-nag/), it fails if the stack passed into `Annotations.fromStack()` is a `ProductStack`. ### Expected Behavior The expectation would be that the test would pass/fail based on the CDK-Nag output/findings. ### Current Behavior The mentioned code will fail with an error similar to: `Unable to find artifact with id "TestStackTestProductStack` ### Reproduction Steps ```typescript import { Annotations, Match } from 'aws-cdk-lib/assertions'; import { App, Aspects, Stack } from 'aws-cdk-lib'; import { AwsSolutionsChecks } from 'cdk-nag'; import { CdkTestProductStack } from '../lib/cdk_test-product-stack'; describe('cdk-nag AwsSolutions Pack', () => { let productStack: ProductStack; let stack: Stack; let app: App; // In this case we can use beforeAll() over beforeEach() since our tests // do not modify the state of the application beforeAll(() => { // GIVEN app = new App(); stack = new Stack(app, 'test-stack'); productStack = new CdkTestProductStack(stack, 'test-productStack'); // WHEN Aspects.of(app).add(new AwsSolutionsChecks()); }); // THEN test('No unsuppressed Warnings', () => { const warnings = Annotations.fromStack(productStack).findWarning( '*', Match.stringLikeRegexp('AwsSolutions-.*') ); expect(warnings).toHaveLength(0); }); test('No unsuppressed Errors', () => { const errors = Annotations.fromStack(productStack).findError( '*', Match.stringLikeRegexp('AwsSolutions-.*') ); expect(errors).toHaveLength(0); }); }); ``` ### Possible Solution _No response_ ### Additional Information/Context _No response_ ### CDK CLI Version 2.69.0 ### Framework Version _No response_ ### Node.js Version 19.8.1 ### OS macOS Ventura 13.2.1 ### Language Typescript ### Language Version _No response_ ### Other information _No response_

[pahud]

Hi @thebillest

It seems this only happens when Annotations.fromStack() on a ProductStack. Is it correct?

Do you have any possible solution or proposed changes?

[thebillest]

According to the ProductStack documentation:

This stack will not be treated as an independent deployment artifact (won't be listed in "cdk list" or deployable through "cdk deploy"), but rather only synthesized as a template and uploaded as an asset to S3

When I go to the aws-cdk-lib/assertions, look at the Annotations class, and work down to the toMessages method (which is what fromStack calls), it appears that it expects to synth and grab artifactIds. These won't work for ProductStack objects.

[wanjacki]

ProductStack works a bit differently from regular stacks: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_servicecatalog.ProductStack.html This stack will not be treated as an independent deployment artifact (won't be listed in "cdk list" or deployable through "cdk deploy"), but rather only synthesized as a template and uploaded as an asset to S3.

Essentially it just generates a CFN template when you synthesize it. I suggest calling a synth, generating the template which should appear in cdk.out, then running testing against that template.

[thebillest]

While the alternative process of synthesizing a ProductStack and performing testing on that works, does it make sense to integrate fromProductStack() on Annotations or to expand the fromStack() method to handle a ProductStack object?

As it currently stands, it's not trivial to render annotations for any ProductStack object.