After pushing 1.5.5 the pipeline is failing at the UpdatePipeline stage - SelfMutate "Error: No stacks match the name(s) dataall-dev-cicd-stack"

MATTT-P commented 1 year ago

Describe the bug

After pushing 1.5.5 the pipeline is failing at the UpdatePipeline stage - SelfMutate with the following error "Error: No stacks match the name(s) dataall-dev-cicd-stack".

Log:

[Container] 2023/07/31 13:01:44 Waiting for agent ping [Container] 2023/07/31 13:01:45 Waiting for DOWNLOAD_SOURCE [Container] 2023/07/31 13:01:46 Phase is DOWNLOAD_SOURCE [Container] 2023/07/31 13:01:46 CODEBUILD_SRC_DIR=/codebuild/output/src2655486164/src [Container] 2023/07/31 13:01:46 YAML location is /codebuild/readonly/buildspec.yml [Container] 2023/07/31 13:01:46 Setting HTTP client timeout to higher timeout for S3 source [Container] 2023/07/31 13:01:46 Processing environment variables [Container] 2023/07/31 13:01:46 No runtime version selected in buildspec. [Container] 2023/07/31 13:01:46 Moving to directory /codebuild/output/src2655486164/src [Container] 2023/07/31 13:01:46 Configuring ssm agent with target id: codebuild:xxx [Container] 2023/07/31 13:01:46 Successfully updated ssm agent configuration [Container] 2023/07/31 13:01:46 Registering with agent [Container] 2023/07/31 13:01:46 Phases found in YAML: 2 [Container] 2023/07/31 13:01:46 INSTALL: 2 commands [Container] 2023/07/31 13:01:46 BUILD: 1 commands [Container] 2023/07/31 13:01:46 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED [Container] 2023/07/31 13:01:46 Phase context status code: Message: [Container] 2023/07/31 13:01:46 Entering phase INSTALL [Container] 2023/07/31 13:01:46 Running command n 16.15.1 installing : node-v16.15.1 mkdir : /usr/local/n/versions/node/16.15.1 fetch : https://nodejs.org/dist/v16.15.1/node-v16.15.1-linux-x64.tar.xz copying : node/16.15.1 installed : v16.15.1 (with npm 8.11.0)

[Container] 2023/07/31 13:01:57 Running command npm install -g aws-cdk@2 npm WARN config global --global, --local are deprecated. Use --location=global instead.

added 1 package, and audited 2 packages in 1s

found 0 vulnerabilities

[Container] 2023/07/31 13:01:59 Phase complete: INSTALL State: SUCCEEDED [Container] 2023/07/31 13:01:59 Phase context status code: Message: [Container] 2023/07/31 13:01:59 Entering phase PRE_BUILD [Container] 2023/07/31 13:01:59 Phase complete: PREBUILD State: SUCCEEDED [Container] 2023/07/31 13:01:59 Phase context status code: Message: [Container] 2023/07/31 13:01:59 Entering phase BUILD [Container] 2023/07/31 13:01:59 Running command cdk -a . deploy dataall-dev-cicd-stack --require-approval=never --verbose [13:01:59] CDK toolkit version: 2.89.0 (build 2ad6683) [13:01:59] Command line arguments: { : [ 'deploy' ], a: '.', app: '.', 'require-approval': 'never', requireApproval: 'never', verbose: 1, v: 1, lookups: true, 'ignore-errors': false, ignoreErrors: false, json: false, j: false, debug: false, ec2creds: undefined, i: undefined, 'version-reporting': undefined, versionReporting: undefined, 'path-metadata': undefined, pathMetadata: undefined, 'asset-metadata': undefined, assetMetadata: undefined, 'role-arn': undefined, r: undefined, roleArn: undefined, staging: true, 'no-color': false, noColor: false, ci: false, all: false, 'build-exclude': [], E: [], buildExclude: [], force: false, f: false, parameters: [ {} ], 'previous-parameters': true, previousParameters: true, logs: true, concurrency: 1, 'asset-prebuild': true, assetPrebuild: true, '$0': '/usr/local/bin/cdk', STACKS: [ 'dataall-dev-cicd-stack' ], 'S-t-a-c-k-s': [ 'dataall-dev-cicd-stack' ] } [13:01:59] merged settings: { versionReporting: true, assetMetadata: true, pathMetadata: true, output: 'cdk.out', app: '.', context: {}, debug: false, requireApproval: 'never', toolkitBucket: {}, staging: true, bundlingStacks: [ '' ], lookups: true, assetPrebuild: true } [13:02:00] Toolkit stack: CDKToolkit [13:02:00] Setting "CDK_DEFAULT_REGION" environment variable to eu-west-1 [13:02:00] Resolving default credentials [13:02:00] Looking up default account ID from STS [13:02:00] Notices refreshed [13:02:00] Failed to store notices in the cache: Error: ENOENT: no such file or directory, open '/root/.cdk/cache/notices.json' [13:02:00] Default account ID: xxx [13:02:00] Setting "CDK_DEFAULT_ACCOUNT" environment variable to xxx [13:02:00] context: { 'aws:cdk:enable-path-metadata': true, 'aws:cdk:enable-asset-metadata': true, 'aws:cdk:version-reporting': true, 'aws:cdk:bundling-stacks': [ '' ] } [13:02:00] --app points to a cloud assembly, so we bypass synth [13:02:00] Notices refreshed

No stacks match the name(s) dataall-dev-cicd-stack [13:02:00] Error: No stacks match the name(s) dataall-dev-cicd-stack at CdkToolkit.validateStacksSelected (/usr/local/lib/node_modules/aws-cdk/lib/index.js:431:3959) at CdkToolkit.selectStacksForDeploy (/usr/local/lib/node_modules/aws-cdk/lib/index.js:431:2861) at async CdkToolkit.deploy (/usr/local/lib/node_modules/aws-cdk/lib/index.js:429:149904) at async exec4 (/usr/local/lib/node_modules/aws-cdk/lib/index.js:504:52657)

[Container] 2023/07/31 13:02:00 Command did not exit successfully cdk -a . deploy dataall-dev-cicd-stack --require-approval=never --verbose exit status 1 [Container] 2023/07/31 13:02:00 Phase complete: BUILD State: FAILED [Container] 2023/07/31 13:02:00 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk -a . deploy dataall-dev-cicd-stack --require-approval=never --verbose. Reason: exit status 1 [Container] 2023/07/31 13:02:00 Entering phase POST_BUILD [Container] 2023/07/31 13:02:00 Phase complete: POST_BUILD State: SUCCEEDED [Container] 2023/07/31 13:02:00 Phase context status code: Message:

How to Reproduce

Pushed the 1.5.5 code into dev. Pipeline started automatically.

Expected behavior

Build completes successfully.

Your project

No response

Screenshots

OS

n/a

Python version

3.7

AWS data.all version

1.5.5

Additional context

No response

dlpzx commented 1 year ago

Hi @MATTT-P , thanks for opening an issue. Let's see, the issue seems to come for an un-existing stack. Can you have a look at the previous stage (Synth)? In Synth the stack gets synthesized, that means, the CDK code creates a CloudFormation template. In UpdatePipeline the template is re-deployed/updated in CloudFormation.

You should check the latest logs in the Synth, to see the output of the synthesis

dlpzx commented 1 year ago

I am trying to reproduce your issue but I don't run into the same situation. We can sync internally to schedule a call and debug together.

MATTT-P commented 1 year ago

Hi @dlpzx We have found the issue to be a hard coded stack name in app.py (dataall\deploy\app.py), the stack name is defined as:

git_branch = git_branch if git_branch != "" else "main"

We needed it to be "dev", and when we move to production we need it to be "prod".

This should be pulled dynamically from the larger section of code:

if not os.environ.get("DATAALL_REPO_BRANCH", None):

Configuration of the branch in first deployment
git_branch = (
    subprocess.Popen(['git', 'branch', '--show-current'], stdout=subprocess.PIPE)
    .stdout.read().decode('utf-8').rstrip('\n')
)
else:

Configuration of the branch in subsequent deployments
git_branch = os.environ.get("DATAALL_REPO_BRANCH")
git_branch = git_branch if git_branch != "" else "main"

But this functionality isn't working in our case so we manually changed it to what we need.

dlpzx commented 1 year ago

Thanks for the response @MATTT-P!, We'll keep an eye on this issue and definitely go back to your findings if we spot it again. For the moment I will close the issue as it cannot be reproduced, it will be reopened if anyone gets into this situation again

data-dot-all / dataall