Open garethjudson opened 1 year ago
Hi @garethjudson. Thanks for the report!
A copy of your cdk.out/manifest.json
would help in debugging this. You can send it to huijbers ~A~T~ amazon.nl
if you are uncomfortable posting it anywhere here.
Thanks!
Hello @garethjudson What do you mean by "v1.81.0 (typescript)" and "given the issue is visible in 1.81.0, but not 1.76.0"? I can see it is neither CDK's (which is 2.81) nor typescript (5.0.4). Was it a typo?
We're also experiencing this issue with cdk 2.81, we rolled back to 2.71 for now. Typescript 4.9.x. Node 18.15.0.
We also have a high # of stacks with a high # of dependencies.
Correct was a typo, should be v2.81.0
I entered the versions correctly into the subsequent fields I have updated the issue text.
Hi @garethjudson. Thanks for the report!
A copy of your
cdk.out/manifest.json
would help in debugging this. You can send it tohuijbers ~A~T~ amazon.nl
if you are uncomfortable posting it anywhere here.Thanks!
I haven't got the cdk.out/manifest.json
from my failed build, unfortunately, sorry about this.
I tried to reproduce subsequently, by upgrading to v2.81.0 again but when I ran the deployment, it passed 🤦♀️ of course it did.
My only thought here is that we were jumping a number of versions. We were upgrading from 2.24.1 to 2.81.0
In this case maybe applying the update 2.24.1 to 2.76.0 fixed the root cause ? I'm not sure if I can offer any more help, but if there is something let me know !
but when I ran the deployment, it passed 🤦♀️ of course it did.
Of course 😆. That's unfortunate. Any intermediary versions you've been using shouldn't make a difference to this issue if it's in the area I think it is in (but of course, this being software, "should" is the operative word here).
For anyone reading running into the same issue, a copy of manifest.json
emailed to the above address would really help a lot in tracking this down. In the mean time I'll see if I can find and print the dependency cycle if the work graph runs into this problem, that should help us make sense of what's going on if it happens again.
I've had someone send me a manifest.json
and it doesn't reproduce on my machine. I wonder if it might be due to a race condition. If the CLI throws this error, and you run while true; do npx cdk deploy --all; done
(bash) does it eventually succeed? (Or gets stuck in a different place)
I have had issues with cdk deploy since 2.80.0. In 2.81.0 I get the error in the OP. In 2.80.0 there's no error - just one of my 5 stacks deploys, nothing happens for the other stacks.
I have a workaround which is this. Say I have 3 stacks: stack3 depends on stack2 and stack2 depends on stack1. If I run:
cdk deploy stack1 cdk deploy stack1 stack2 stack3
then all of the stacks will deploy successfully. Similarly I've seen that this works:
cdk deploy stack1 cdk deploy --all
This works in 2.80.0 and 2.81.0. I also don't see any errors or unusual behaviour in 2.79.0 so I suspect the defect originates in 2.80.0.
I've identified the problem. The next release will have the fix.
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
@rix0rrr just tried with the changes from #25846 (using patch-package
) and I'm still getting the error Unable to make progress anymore among: ...
Thanks for testing @jogold! That's annoying. Do you have anything you can share so that I can repro?
Thanks for testing @jogold! That's annoying. Do you have anything you can share so that I can repro?
Will send you my manifest.json
.
And your *.asset.json
files as well please.
And your
*.asset.json
files as well please.
sent
Thanks! Unfortunately it didn't help, with the code above there are no cycles anymore in the graph that's built from your manifest.json
(or at least, as far as I could tell). Can you send me the error message as well? It should have the cycle that it found.
@rix0rrr trying now with v2.83.0, will let you know if this can be closed.
@rix0rrr the issue is fixed for me in v2.83.0. Thanks and sorry again for the confusion with the patch.
Describe the bug
Mid way through a cdk deployment the following error is raised:
❌ Deployment failed: Error: Unable to make progress anymore among: <...> at WorkGraph.updateReadyPool (/builds/.../node_modules/aws-cdk/lib/index.js:400:138218) at start (/builds/.../node_modules/aws-cdk/lib/index.js:400:136061) at /builds/.../node_modules/aws-cdk/lib/index.js:400:136488 at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)
Unable to make progress anymore among:
<...>
Where '<...>' is a printed list of every stack, simliar to: wwwStack := completed stack, ... yyyStack:= pending asset-publish, wwwStack := pending stack
This issue occurs on v2.81.0 (typescript) but downgrading to 2.76.0 resolves the issue. There is no detail to determine a cause
Expected Behavior
Either the stack deploys successfully or an error message with a clear description of the cause is displayed
Current Behavior
Deployment of stacks is partially completed, and then the deployment fails with:
❌ Deployment failed: Error: Unable to make progress anymore among: <...>
Reproduction Steps
I don't know how to, other than by using my environment deployment, because I can't determine the cause of the issue. Given our environment is ~50 cdk stacks, with a relatively straight forward dependency order, I can't determine a cause. Other than to deploy my stack.
Happy to provide more detail if I can get help with how best to do so. Suspect it's related to dependant stacks, but I have no idea.
Possible Solution
Downgrading to version 2.76.0
Additional Information/Context
I get an issue in v2.81.0 of cdk (with typescript), which I suspect is related to dependant stacks.
In my environment we have many stacks which are dependant. This was making a linear dependency where each stack was dependant on an immediately prior stack.
Found this easiest way to manage dependency order, where you individual devs may not have the context to determine this easily. This way we don't have to think about dependency order too hard, earlier means before anything later...easy. If this were causing the issue, could change it, however, before expending development effort, would like to understand the cause. This issue looks suspiciously similar to https://github.com/aws/aws-cdk/issues/25714, given the issue is visible in 2.81.0, but not 2.76.0
Downgrading to version 2.76.0 resolves the issue. Happy to help with any additional information.
Thanks a bunch.
CDK CLI Version
2.81.0 (build bd920f2)
Framework Version
2.81.0
Node.js Version
18.16.0
OS
macOS , alpine linux (node:hydrogen-alpine container)
Language
Typescript
Language Version
5.0.4
Other information
Could be related to https://github.com/aws/aws-cdk/issues/25714
Commented on that issue, but realised the comment was at the end of a closed issue, so thought I better raise a new one.