(cdk-assets): ecs.RepositoryImage.fromTarball not working in cdk pipelines

TheFlexican commented 2 years ago

What is the problem?

Hi, when trying to create ecs cluster with ecs.RepositoryImage.fromTarball the Asset build step in my pipeline fails with the error: Error parsing reference: "" is not a valid repository/tag: invalid reference format

The command generated to load the docker image in the Asset build step is not properly quoted. Log output of the Asset build step. Building Docker image using command 'sh,-c,docker load -i asset.<random numbers>.tar | sed "s/Loaded image: //g"'

This turns into something like below and that won't work, as sh -c is not taking the additional arguments passed on the command line into account. If you run the command below you get the help output show of docker, instead of docker loading the image. sh -c docker load -i asset.<random numbers>.tar | sed "s/Loaded image: //g

The command should be properly quoted for this to work like below: sh -c 'docker load -i asset.<random numbers>.tar | sed "s/Loaded image: //g"'

Reproduction Steps

Create a pipeline with aws-cdk-lib/pipelines, add a stage with an ECR stack that contains the ecs.RepositoryImage.fromTarball, and provide it with a local docker image.

What did you expect to happen?

Ecs cluster being created

What actually happened?

Pipeline execution stopped.

CDK CLI Version

2.1.0 (build f4f18b1)

Framework Version

No response

Node.js Version

v16.3.0

OS

Ubuntu

Language

Typescript

Language Version

No response

Other information

I tried patching the tarball-asset.js to include additional quotes, and tried every possible quote combination I could think of but none of them worked. Then I found out that shell.js is doing escaping of characters x = x.replace("'", "'\"'\"'"); so I sort of gave up and created this issue.

derrike commented 2 years ago

I just had this issue myself and I don't think the issue is related to the quotes. The problem is the path that it is expected to find the tar. The cdk-assets is expecting the tar to be in the assembly CDK Pipeline folder, but that folder just has *.json files. The CDK Pipelines creates additional sub directories, so the assets are actually one level up.

I updated the executable line to go up one directory and manually ran cdk-assets to publish, which worked:

"executable": [
  "sh",
  "-c",
  "docker load -i ../asset.<random_id>.tar | sed \"s/Loaded image: //g\""
]

^ docker load -i ../

I would think that this is a problem related to the way CDK Pipelines is generating the asset file for the from_tarball. It needs to be updated to correctly point up one parent directory.

AndrewGuenther commented 2 years ago

The new issue name isn't accurate. This isn't broken regardless to whether you're using CDK pipelines or not.

AndrewGuenther commented 2 years ago

My issue may actually be different, but the fix could be the same. I filed #18822 separately since the repro steps are a bit different.

TheFlexican commented 2 years ago

This is still happening as of today with cdk version 2.12.0 using cdk pipelines.

Below is a recent build log.

[Container] 2022/02/16 14:32:05 Phase complete: INSTALL State: SUCCEEDED
[Container] 2022/02/16 14:32:05 Phase context status code:  Message: 
[Container] 2022/02/16 14:32:05 Entering phase PRE_BUILD
[Container] 2022/02/16 14:32:05 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2022/02/16 14:32:05 Phase context status code:  Message: 
[Container] 2022/02/16 14:32:05 Entering phase BUILD
[Container] 2022/02/16 14:32:05 Running command cdk-assets --path "assembly-XXXXXXXX/XXXXXXXXEcsStack1352F32B.assets.json" --verbose publish "2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351:111111111111-eu-west-1"
verbose: Loaded manifest from assembly-XXXXXXXX/XXXXXXXXEcsStack1352F32B.assets.json: 2 assets found
verbose: Applied selection: 1 assets selected.
info   : [0%] start: Publishing 2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351:111111111111-eu-west-1
verbose: [0%] check: Check 111111111111.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-111111111111-eu-west-1:2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351
verbose: [0%] debug: docker login --username AWS --password-stdin https://111111111111.dkr.ecr.eu-west-1.amazonaws.com
verbose: [0%] build: Building Docker image using command 'sh,-c,docker load -i asset.2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351.tar | sed "s/Loaded image: //g"'
verbose: [0%] upload: Push 111111111111.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-111111111111-eu-west-1:2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351
verbose: [0%] debug: docker tag  111111111111.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-111111111111-eu-west-1:2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351
Error parsing reference: "" is not a valid repository/tag: invalid reference format
error  : [100%] fail: docker tag  111111111111.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-111111111111-eu-west-1:2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351 exited with error code 1: Error parsing reference: "" is not a valid repository/tag: invalid reference format
Failure: Error: docker tag  111111111111.dkr.ecr.eu-west-1.amazonaws.com/cdk-hnb659fds-container-assets-111111111111-eu-west-1:2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351 exited with error code 1: Error parsing reference: "" is not a valid repository/tag: invalid reference format
    at ChildProcess.<anonymous> (/usr/local/lib/node_modules/cdk-assets/lib/private/shell.js:46:24)
    at Object.onceWrapper (events.js:482:26)
    at ChildProcess.emit (events.js:375:28)
    at ChildProcess.emit (domain.js:470:12)
    at maybeClose (internal/child_process.js:1055:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
[Container] 2022/02/16 14:32:05 Command did not exit successfully cdk-assets --path "assembly-XXXXXXXX/XXXXXXXXEcsStack1352F32B.assets.json" --verbose publish "2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351:111111111111-eu-west-1" exit status 1
[Container] 2022/02/16 14:32:05 Phase complete: BUILD State: FAILED
[Container] 2022/02/16 14:32:05 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk-assets --path "assembly-XXXXXXXX/XXXXXXXXEcsStack1352F32B.assets.json" --verbose publish "2d460b553524fa2bc059e61223b9b36fc086b9db8558ca3a7fcd75be26f33351:111111111111-eu-west-1". Reason: exit status 1
[Container] 2022/02/16 14:32:05 Entering phase POST_BUILD
[Container] 2022/02/16 14:32:05 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2022/02/16 14:32:05 Phase context status code:  Message:

TheFlexican commented 2 years ago

As said by @derrike it's indeed the path that is incorrect.

I fixed it temporary by using sed in the buildspec file for the docker asset.

version: '0.2'
phases:
  install:
    commands:
    - npm install -g cdk-assets@2
  build:
    commands:
      - sed -i 's|asset\.|../asset.|g' assembly-xxxxxxx/xxxxxxx.assets.json
      - cdk-assets --path "assembly-xxxxxxx/xxxxxxx.assets.json" --verbose publish "xxxxxxxxxxxx:xxxxxxxxxxxx-eu-west-1"

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

Richardmbs12 commented 1 year ago

Good day,

This issue was never resolved? I still get this on 2.81.0?

I know theres a workaround mentioned here, but its really not ideal as I dont want to edit the buildspec via the console and not have my source of truth be my cdk stack or pipeline...?

aws / aws-cdk