Open elderapo opened 6 months ago
Thanks for the bug report @elderapo. We'll keep an eye out for this to see if we can track it down. In the meantime, if you do find a consistent reproduction, please let us know!
This is hitting me as well in GHE Workflow Action actions/pulumi-actions@v4
"dependencies": {
"@pulumi/aws": "^6.0.0",
"@pulumi/awsx": "^2.0.2",
"@pulumi/pulumi": "^3.113.0",
"typescript": "^5.0.0"
}
For me v4 tag is tagged this Jun 5, 2023 commit/4204b4e8a7e703da96ba5dd4c3a667adeee35812
which looks to be v4.4
In my case I have two new docker.Image(...)
instances in the same stack, each building a different Dockerfile
I need the .repoDigest
from both so I can do a follow up deploy.
But this fails intermittently as the .repoDigest
is <none>@<none>
which is not an acceptable input to my FargetTaskDefinition.
What's particularly odd, is that one of the new docker.Image(...)
instances produces it's repoDigest correctly.
I had this happen earlier this week, but of the two images they flipped which one was <none>@<none>
The first time It happened I resorted to commenting out the declaration of the offending image, building, then, uncommenting out, then building again.
Just now, I tried...
Resources:
4 unchanged
So that didn't help, pulumi stack output
still has one of the two images as none@none
Outputs:
- ghApiExporterImage : "<none>@<none>"
yes to that,
Outputs:
+ ghApiExporterImage : "<none>@<none>"
up
deleted the resource and output.A couple thoughts... Obviously it'd be great if this just didn't break, but when it does break the fact that I have to change the code, run pulumi, revert the code, run pulumi makes for a real head ache.
I wonder, would a pinpoint state delete command, followed by a up
, be the way to go here (until you fix it that is) ?
@alfred-stokespace If you can reproduce it on demand, please create a simple repro repo. It should help the Pulumi team to get to the bottom of this issue. It only happened to me a couple of times around the time I opened this issue.
Meanwhile, I am using this trick to prevent accidental deployments when images don't build successfully:
const image = new docker.Image(...);
const validatedImage = image.repoDigest.apply(digest => {
if (digest === "<none>@<none>") {
throw new ResourceError(
`Digest(${digest}) is "<none>@<none>"! Image either failed to build or push...`,
image,
true,
);
}
/**
* Possibilities:
* sha256:xxx
* docker.io/user/repo@sha256:xxx
*/
if (!digest.includes("sha256:")) {
throw new ResourceError(
`Digest(${digest}) does not include sha256 prefix! Image either failed to build or push...`,
image,
true,
);
}
return digest;
});
What happened?
I recently upgraded
@pulumi/docker
from v3 to v4. Everything seemed okay until a couple of hours ago.It appears that pulumi docker either failed to build the image (got stuck at some step?, I extensively use docker image multi-step builds), failed to push it to the docker registry but still successfully went through preview (
buildOnPreview
was set totrue
) and then deployed changes to the k8s cluster. What ended up happening is a broken deployment where some pods (not all) had their images set to<none>@<none>
.Example
Unfortunately, I am unable to provide an example/reproduce the bug however there are snippets of how
DockerProvider
/Image
instances were constructed:Output of
pulumi about
I am unable to get output of
pulumi about
because the issue in question occurred in github action (and I no longer access the VM instance).Pulumi version was
3.107.0
, installed throughpulumi/actions@v5
(SHA:76683de37aa44910871ba6cef36557780f2e41d1
) OS: Ubuntu 22.04Additional context
I suspect the issue might've been caused by temporary network issues between "dedicated docker image builder server" and github action VM (where Pulumi is executed).