awslabs / landing-zone-accelerator-on-aws

Deploy a multi-account cloud foundation to support highly-regulated workloads and complex compliance requirements.
https://aws.amazon.com/solutions/implementations/landing-zone-accelerator-on-aws/
Apache License 2.0
565 stars 453 forks source link

Unable to fetch parameters [/accelerator/network/vpc/Shared-Network/id] from parameter store for this account #177

Closed jacekhewko closed 1 year ago

jacekhewko commented 1 year ago

Hello,

AWSAccelerator-Pipeline fails on Network_Associations step with the below:


AWSAccelerator-NetworkAssociationsGwlbStack-xxx-eu-west-2: building assets...
--
563 |  
564 | Assuming role arn:aws:iam::xxx:role/AWSControlTowerExecution for 3600 seconds
565 | credentials returned by plugin 'cdk-assume-role-plugin' could not be used to assume 'arn:aws:iam::xxx:role/cdk-accel-deploy-role-xxx-eu-west-2', but are for the right account. Proceeding anyway.
566 |  
567 | ❌  AWSAccelerator-NetworkAssociationsStack-xxx-eu-west-2 failed: Error [ValidationError]: Unable to fetch parameters [/accelerator/network/vpc/Shared-Network/id] from parameter store for this account.
568 | at Request.extractError (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/protocol/query.js:50:29)
569 | at Request.callListeners (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
570 | at Request.emit (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
571 | at Request.emit (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/request.js:686:14)
572 | at Request.transition (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/request.js:22:10)
573 | at AcceptorStateMachine.runTo (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/state_machine.js:14:12)
574 | at /codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/state_machine.js:26:10
575 | at Request.<anonymous> (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/request.js:38:9)
576 | at Request.<anonymous> (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/request.js:688:12)
577 | at Request.callListeners (/codebuild/output/src050/src/s3/00/source/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
578 | code: 'ValidationError',
579 | time: 2023-06-15T10:56:59.485Z,
580 | requestId: '86c77412-1e95-4d2d-83bb-83ac8ad1e4b1',
581 | statusCode: 400,
582 | retryable: false,
583 | retryDelay: 916.6971816393245
584 | }
585 |  
586 | ❌ Deployment failed: Error: Stack Deployments Failed: ValidationError: Unable to fetch parameters [/accelerator/network/vpc/Shared-Network/id] from parameter store for this account.
587 | at deployStacks (/codebuild/output/src050/src/s3/00/source/node_modules/aws-cdk/lib/deploy.ts:61:11)
588 | at runMicrotasks (<anonymous>)
589 | at processTicksAndRejections (node:internal/process/task_queues:96:5)
590 | at async CdkToolkit.deploy (/codebuild/output/src050/src/s3/00/source/node_modules/aws-cdk/lib/cdk-toolkit.ts:339:7)
591 | at async Function.execute (/codebuild/output/src050/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/toolkit.ts:312:9)
592 | 2023-06-15 10:56:59.494 \| error \| toolkit \| Stack Deployments Failed: ValidationError: Unable to fetch parameters [/accelerator/network/vpc/Shared-Network/id] from parameter store for this account.
593 | 2023-06-15 10:56:59.494 \| error \| toolkit \| Deployment failed
594 | 2023-06-15 10:56:59.499 \| error \| accelerator \| Runtime Error
595 | /codebuild/output/src050/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/accelerator.ts:56
596 | throw new Error('Synthesis failed');
597 | ^
598 |  
599 | Error: Synthesis failed
600 | at process.<anonymous> (/codebuild/output/src050/src/s3/00/source/packages/@aws-accelerator/accelerator/lib/accelerator.ts:41:9)
601 | at process.emit (node:events:525:35)
602 | at process.emit (node:domain:489:12)
603 | at process.emit.sharedData.processEmitHook.installedValue [as emit] (/codebuild/output/src050/src/s3/00/source/node_modules/@cspotcode/source-map-support/source-map-support.js:745:40)
604 | at process._fatalException (node:internal/process/execution:149:25)
605 | error Command failed with exit code 7.
606 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
607 |  
608 | [Container] 2023/06/15 10:56:59 Command did not exit successfully yarn run ts-node --transpile-only cdk.ts --require-approval never $CDK_OPTIONS --config-dir $CODEBUILD_SRC_DIR_Config --partition aws --app cdk.out exit status 7
609 | [Container] 2023/06/15 10:56:59 Phase complete: BUILD State: FAILED
610 | [Container] 2023/06/15 10:56:59 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: yarn run ts-node --transpile-only cdk.ts --require-approval never $CDK_OPTIONS --config-dir $CODEBUILD_SRC_DIR_Config --partition aws --app cdk.out. Reason: exit status 7
611 | [Container] 2023/06/15 10:56:59 Entering phase POST_BUILD
612 | [Container] 2023/06/15 10:56:59 Phase complete: POST_BUILD State: SUCCEEDED
613 | [Container] 2023/06/15 10:56:59 Phase context status code:  Message:

This happened when added 6 new AWS accounts into accounts-config.yaml. Nothing else have changed.

Anyone got an idea why would this happen and how to fix this?

Thanks, Jacek

erwaxler commented 1 year ago

Hi @jacekhewko , thank you for your interest in the Landing Zone Accelerator on AWS! I have been able to reproduce this error in my environment, and the team is currently working on a fix.

The root cause of the problem is that SSM Parameters are not created in new AWS Accounts as they should be. In your case, the SSM parameter that was not created was /accelerator/network/vpc/Shared-Network/id. As an immediate workaround, you can manually create the missing SSM Parameter(s) in the new account. The correct value should be stored at the same path in SSM Parameter Store within a preexisting account.

I will update this issue as the team makes progress on implementing the fix. Thank you again for identifying this bug!

FlemmingBehrend commented 1 year ago

+1

erwaxler commented 1 year ago

Hi @jacekhewko @FlemmingBehrend , thank you for your patience on this issue. This behavior was fixed in our latest release v1.4.2, please update your LZA deployment to version 1.4.2 in order to resolve the behavior. I'll be closing this issue now that the fix has been released, thank you again for bringing this to our attention.

jacekhewko commented 1 year ago

Hi @erwaxler,

Thanks for the info.

Unfortunately, the error did not disappear after upgrading to v1.4.2 and launching the pipeline again, nor can I find any /accelerator/network/vpc/Shared-Network/id parameter in the main or the child AWS accounts of the solution. Thus I am unable to get rid of the pipeline failure.

erwaxler commented 1 year ago

Hi @jacekhewko , you will need to create those parameters manually for this pipeline run only. Please see the v1.4.2 Release Notes for more detailed instructions on which SSM parameters need to be created.

I would recommend the above approach. If this is a development environment that can tolerate downtime, another option would be to unshare then reshare the subnet to have automatically create the parameters. Regardless of the path you take, please let me know if this resolves your error.

awsclemj commented 1 year ago

Hello,

We haven't heard from you on this issue in a while. If you require any additional assistance, please feel free to reply and/or re-open this issue.

Thank you for your interest in Landing Zone Accelerator on AWS!