Open solomonshorser opened 1 year ago
Hi @solomonshorser, thank you for reporting this. To help with debugging, would it be possible for you to attach the complete output of pulumi up
with increased log level? Thank you!
That's a lot of output (> 1000 lines!), but when I turned logging up to 6, I found this snippet, which seems relevant:
I1115 10:33:43.679106 32847 step_executor.go:389] StepExecutor worker(10): applying step delete-replaced on urn:pulumi:my-stack-dev::my-stack::google-native:iam/v1:Provider::aws_identity_pool-provider (preview false)
I1115 10:33:43.884428 32847 state.go:304] error copying .pulumi/stacks/my-stack-dev.json.gz to .pulumi/stacks/my-stack-dev.json.gz.bak: blob (key ".pulumi/stacks/my-stack-dev.json.gz -> .pulumi/stacks/my-stack-dev.json.gz.bak") (code=NotFound): googleapi: Error 404: No such object: pulumi-bucket/.pulumi/stacks/my-stack-dev.json.gz, notFound
I1115 10:33:43.936311 32847 state.go:310] error deleting source object after rename: .pulumi/stacks/my-stack-dev.json.gz (blob (key ".pulumi/stacks/my-stack-dev.json.gz") (code=NotFound): storage: object doesn't exist) skipping
I1115 10:33:44.330619 32847 deployment_executor.go:231] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
I1115 10:33:44.330658 32847 deployment_executor.go:418] deploymentExecutor.handleSingleEvent(...): received RegisterResourceEvent
I1115 10:33:44.330627 32847 source_eval.go:1393] ResourceMonitor.RegisterResourceOutputs operation finished: urn=urn:pulumi:my-stack-dev::my-stack::AwsGcpIdentityPoolBinding::awsbindings, #outs=2
I1115 10:33:44.332547 32847 eventsink.go:59] RegisterResourceOutputs RPC finished: urn=urn:pulumi:my-stack-dev::my-stack::AwsGcpIdentityPoolBinding::awsbindings; err: null, resp:
I1115 10:33:44.333217 32847 step_generator.go:269] Checking step update for urn:pulumi:my-stack-dev::my-stack::google-native:cloudresourcemanager/v3:ProjectIamPolicy::b056e491eba9_iam_policy
I1115 10:33:44.333259 32847 step_executor.go:389] StepExecutor worker(-2): worker received chain for execution
I1115 10:33:44.333275 32847 step_executor.go:389] StepExecutor worker(-2): worker waiting for incoming chains
I1115 10:33:44.333303 32847 step_executor.go:389] StepExecutor worker(11): launching oneshot worker
I1115 10:33:44.333402 32847 eventsink.go:59] RegisterResourceOutputs RPC finished: urn=urn:pulumi:my-stack-dev::my-stack::AwsGcpIdentityPoolBinding::awsbindings; err: null, resp:
I1115 10:33:44.536002 32847 state.go:304] error copying .pulumi/stacks/my-stack-dev.json.gz to .pulumi/stacks/my-stack-dev.json.gz.bak: blob (key ".pulumi/stacks/my-stack-dev.json.gz -> .pulumi/stacks/my-stack-dev.json.gz.bak") (code=NotFound): googleapi: Error 404: No such object: pulumi-bucket/.pulumi/stacks/my-stack-dev.json.gz, notFound
I1115 10:33:44.585400 32847 state.go:310] error deleting source object after rename: .pulumi/stacks/my-stack-dev.json.gz (blob (key ".pulumi/stacks/my-stack-dev.json.gz") (code=NotFound): storage: object doesn't exist) skipping
I1115 10:33:44.893330 32847 eventsink.go:86] eventSink::Error(<{%reset%}>error sending request: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 404 (Not Found)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>404.</b> <ins>That’s an error.</ins>
<p>The requested URL <code>/aws_identity_pool-provider?alt=json</code> was not found on this server. <ins>That’s all we know.</ins>
<{%reset%}>)
The message about the my-stack-dev.json.gz
file being missing seems strange. It's true that the file is not there, but I looked in the bucket where all of the stacks JSON files get stored, and there are no *.gz
files for any of the other stacks. I guess the question is: Why is Pulumi concerned that this file is missing for this stack, but has never complained about it for other stacks?
With logging level 9, I got a LOT more information. I was able to find the log entries with the full URL:
I1115 11:02:20.621972 41504 state.go:310] error deleting source object after rename: .pulumi/stacks/my-stack-dev.json.gz (blob (key ".pulumi/stacks/my-stack-dev.json.gz") (code=NotFound): storage: object doesn't exist) skipping
I1115 11:02:20.979627 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>I1115 11:02:20.979578 41576 http.go:107] HTTP Request Begin DELETE https://iam.googleapis.com/aws_identity_pool-provider?alt=json ===================================================
<{%reset%}>)
I1115 11:02:20.979684 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>DELETE /aws_identity_pool-provider?alt=json HTTP/1.1
<{%reset%}>)
I1115 11:02:20.979702 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Host: iam.googleapis.com
<{%reset%}>)
I1115 11:02:20.979716 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Content-Type: application/json
<{%reset%}>)
I1115 11:02:20.979726 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>User-Agent: Pulumi/3.7.1 (GPN:Pulumi; https://www.pulumi.com) pulumi-google-native/v0.7.0
<{%reset%}>)
I1115 11:02:20.979737 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Request End DELETE https://iam.googleapis.com/aws_identity_pool-provider?alt=json
<{%reset%}>)
I1115 11:02:21.047600 41504 state.go:261] Saved stack my-stack-dev checkpoint to: .pulumi/stacks/my-stack-dev.json (backup=.pulumi/stacks/my-stack-dev.json.bak)
I1115 11:02:21.047672 41504 deployment_executor.go:231] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
This is followed by over 1000 lines that are mostly logging (un)marshalling activity. Then there is this:
I1115 11:02:21.456410 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>I1115 11:02:21.456364 41576 http.go:115] HTTP Response Begin DELETE [https://iam.googleapis.com/aws_identity_pool-provider?alt=json ===================================================
<{%reset%}>)
I1115 11:02:21.456447 41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>HTTP/2.0 404 Not Found
Thank you for going through all these logs, @solomonshorser!
When the error occurs, does the provider my_aws_identity_pool-provider
exist in GCP or not? Did you check via the UI, for instance? That could tell us if Pulumi is using a wrong URL or whether the delete operation shouldn't be happening at all.
Also, is it possible you made changes to your infra outside of Pulumi? In that case, pulumi refresh
can help to get the state back in sync.
The object exists in GCP, according to the UI. I can try to run refresh
again, though I think I did that before and it didn't really resolve much.
What happened?
Pulumi tried to replace a
google-native:iam/v1:Provider
object. The operation failed with an error from Google stating that a URL could not be found.Steps to reproduce
google-native:iam/v1:Provider
for agoogle-native:iam/v1:WorkloadIdentityPool
.pulumi up
, say "yes" to complete the operation, when Pulumi suggests replacing thegoogle-native:iam/v1:Provider
.Expected Behavior
I expected Pulumi to update the object properly.
Actual Behavior
Pulumi produced an error. It looks like Pulumi tried to access a URL that does not exist in GCP.
Output of
pulumi about
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).