python/aws_cdk: synth lock file not released on python subprocess error

[bishax-pp]

Describe the bug

When cdk synth is run and the underlying Python code raises an Exception, e.g. due to a syntax error, the next run of synth fails with:

Another CLI (PID=1) is currently synthing to cdk.out. Invoke the CLI in sequence, or use '--output' to synth into different directories.

This appears to happen because the lock file in cdk.out/synth.lock is not released.

This is normally not noticed because the PID in the lock file is no longer live - https://github.com/aws/aws-cdk/blob/6d581d7d4a0107dc5acd532c603f90d253d8f72e/packages/aws-cdk/lib/api/util/rwlock.ts#L7 ; however when running CDK within a docker container then PIDs reoccur

Expected Behavior

cdk.out/synth.lock is released

Current Behavior

cdk.out/synth.lock is not released causing Another CLI (PID=1) is currently synthing to cdk.out. Invoke the CLI in sequence, or use '--output' to synth into different directories. to be raised on the next synth command unless the lock file is manually removed.

Reproduction Steps

1. Add assert 0 or something that will raise an exception in a Python CDK app
2. Run cdk synth for that app within docker
3. Run cdk synth again within docker and observe the issue because 2 and 3 have the same PID

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.72.1

Framework Version

No response

Node.js Version

18.13.0

OS

Debian GNU/Linux 11

Language

Python

Language Version

No response

Other information

No response

[pahud]

Hi @bishax-pp

Does it only happen when you cdk synth in the docker container? Can you share your use case why you need to do that in the container?

[oliviagunton]

In case it helps to track down the cause, we repro this with v2.61 and higher of the CDK CLI and not with v2.60 or below. (Haven't yet checked patch versions between v2.60.0 and v2.61.0 to narrow it down further.)

[bishax-pp]

Hi @pahud

Yes it has only happened to me for CDK in docker; however I believe that it would occur if:

• I ran a failing cdk synth (with PID x) outside of docker
• Rebooted my machine
• Ran failing cdk synth again and by chance the PID was x again

We use cdk in docker because it makes managing/updating our environments easier.

[pahud]

@bishax-pp

Can you share more about your Dockerfile and how you run the docker so we can build the image and run it in our account to reproduce this?

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[bishax-pp]

Hey @pahud

The dockerfile looks something like

ARG PYTHON_VERSION
FROM --platform=linux/amd64 python:$PYTHON_VERSION-slim-bullseye

RUN apt-get update && \
    apt-get install -y --no-install-recommends npm wget

RUN useradd -s /bin/bash -m user

WORKDIR /home/user

RUN npm install -g n
RUN n stable

USER user
ARG CDK_VERSION
RUN npm install aws-cdk@$CDK_VERSION
ENV PATH=$PATH:/home/user/node_modules/aws-cdk/bin/

USER root
# omitted: install python dependencies and copy in source

USER user
ENTRYPOINT ["cdk"]

[CameronLofy]

I only fixed this by manually deleting the .lock file. I was able to do cdk synth after that.

[LudwigHoff]

Bump pls? Thank you! :-)

v2.82

(.net cdk but same-same behaviour)

[leonk-sportsbet]

Still happens with cdk 2.87.0:

When mounting local FS in the docker:

docker run -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN --user $(id -u):$(id -g) --rm -it -v $(pwd):/app -w /app node:18-bullseyecustom /bin/bash

Dockerfile:

FROM node:18.16.1-bullseye-slim

# To avoid frontend dockerfile.v0 error, export vars on cki before image build:
# https://medium.com/code-kings/docker-how-to-fix-failed-to-solve-with-frontend-dockerfile-v0-error-when-building-a-docker-image-6d7dc95abd27
# export DOCKER_BUILDKIT=0
# export COMPOSE_DOCKER_CLI_BUILD=0

# Timezone config
RUN cp /usr/share/zoneinfo/Australia/Melbourne /etc/localtime && \
    echo "Australia/Melbourne" >  /etc/timezone && \
    dpkg-reconfigure --frontend noninteractive tzdata

# Using entrypoint script to prevent 'REPL session history will not be persisted' err
WORKDIR /app
COPY entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh
RUN apt-get update && apt-get install -y \
        git \
        && rm -rf /var/lib/apt/lists/*

# Deal with "Your cache folder contains root-owned files, due to a bug in/previous versions of npm which has since been addressed" error
RUN mkdir -p /.npm && chmod 777 /.npm

ENTRYPOINT ["docker-entrypoint.sh"]

CMD [ "node" ]

error:

Another CLI (PID=54) is currently synthing to cdk.out. Invoke the CLI in sequence, or use '--output' to synth into different directories.

Re-running synth again solves an issue but it will definitely be a problem in cicd! Any ideas on how to avoid it?

[trompx]

Also happening to me in 2.100.0. Removing the cdk.out/synth.lock manually works but have to do it every time.

[kmjain]

Does anyone has any solution to this problem? I am using Jenkins pipeline to deploy cdk stacks, and using python dockerfile… and facing this issue, not sure what’s the best solution.. any help is appreciated, changing the output directory is painful, thanks!

[vinhtranvuhai]

Does anyone has any solution to this problem? I am using Jenkins pipeline to deploy cdk stacks, and using python dockerfile… and facing this issue, not sure what’s the best solution.. any help is appreciated, changing the output directory is painful, thanks!

In my case, simply deleting the output file resolved the issue. Everything is working now

[jtele2]

+1 I'm still having this issue. Not in a docker container either.