search

pulumi / pulumi-google-native

Apache License 2.0
71 stars 18 forks source link

Google API URL not found #743

Open solomonshorser opened 1 year ago

solomonshorser commented 1 year ago

What happened?

Pulumi tried to replace a google-native:iam/v1:Provider object. The operation failed with an error from Google stating that a URL could not be found.

Steps to reproduce

  1. Create a google-native:iam/v1:Provider for a google-native:iam/v1:WorkloadIdentityPool.
  2. Run pulumi up, say "yes" to complete the operation, when Pulumi suggests replacing the google-native:iam/v1:Provider.
  3. Observe an error.

Expected Behavior

I expected Pulumi to update the object properly.

Actual Behavior

Pulumi produced an error. It looks like Pulumi tried to access a URL that does not exist in GCP. 

Diagnostics:
  google-native:iam/v1:Provider (my_aws_identity_pool-provider):
    error: error sending request: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
    <html lang=en>
      <meta charset=utf-8>
      <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
      <title>Error 404 (Not Found)!!1</title>
      <style>
[SNIP...]
      </style>
      <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
      <p><b>404.</b> <ins>That’s an error.</ins>
      <p>The requested URL <code>/my_aws_identity_pool-provider?alt=json</code> was not found on this server.  <ins>That’s all we know.</ins>

  pulumi:pulumi:Stack (my-stack):
    error: update failed

Output of pulumi about

CLI
Version      3.46.1
Go Version   go1.19.2
Go Compiler  gc

Plugins
NAME           VERSION
gcp            6.42.0
google-native  0.27.0
google-native  0.7.0
nodejs         unknown
random         4.8.2

Host
OS       darwin
Version  12.6.1
Arch     x86_64

This project is written in nodejs: executable='/usr/local/bin/node' version='v19.0.1'

...

Dependencies:
NAME                                  VERSION
@pulumi/gcp                           6.42.0
@pulumi/google-native                 0.27.0
@pulumi/pulumi                        3.46.1
@types/node                           10.17.60
@google-cloud/bigquery                6.0.3

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

thomas11 commented 1 year ago

Hi @solomonshorser, thank you for reporting this. To help with debugging, would it be possible for you to attach the complete output of pulumi up with increased log level? Thank you!

solomonshorser commented 1 year ago

That's a lot of output (> 1000 lines!), but when I turned logging up to 6, I found this snippet, which seems relevant:

I1115 10:33:43.679106   32847 step_executor.go:389] StepExecutor worker(10): applying step delete-replaced on urn:pulumi:my-stack-dev::my-stack::google-native:iam/v1:Provider::aws_identity_pool-provider (preview false)
I1115 10:33:43.884428   32847 state.go:304] error copying .pulumi/stacks/my-stack-dev.json.gz to .pulumi/stacks/my-stack-dev.json.gz.bak: blob (key ".pulumi/stacks/my-stack-dev.json.gz -> .pulumi/stacks/my-stack-dev.json.gz.bak") (code=NotFound): googleapi: Error 404: No such object: pulumi-bucket/.pulumi/stacks/my-stack-dev.json.gz, notFound
I1115 10:33:43.936311   32847 state.go:310] error deleting source object after rename: .pulumi/stacks/my-stack-dev.json.gz (blob (key ".pulumi/stacks/my-stack-dev.json.gz") (code=NotFound): storage: object doesn't exist) skipping
I1115 10:33:44.330619   32847 deployment_executor.go:231] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)
I1115 10:33:44.330658   32847 deployment_executor.go:418] deploymentExecutor.handleSingleEvent(...): received RegisterResourceEvent
I1115 10:33:44.330627   32847 source_eval.go:1393] ResourceMonitor.RegisterResourceOutputs operation finished: urn=urn:pulumi:my-stack-dev::my-stack::AwsGcpIdentityPoolBinding::awsbindings, #outs=2
I1115 10:33:44.332547   32847 eventsink.go:59] RegisterResourceOutputs RPC finished: urn=urn:pulumi:my-stack-dev::my-stack::AwsGcpIdentityPoolBinding::awsbindings; err: null, resp: 
I1115 10:33:44.333217   32847 step_generator.go:269] Checking step update for urn:pulumi:my-stack-dev::my-stack::google-native:cloudresourcemanager/v3:ProjectIamPolicy::b056e491eba9_iam_policy
I1115 10:33:44.333259   32847 step_executor.go:389] StepExecutor worker(-2): worker received chain for execution
I1115 10:33:44.333275   32847 step_executor.go:389] StepExecutor worker(-2): worker waiting for incoming chains
I1115 10:33:44.333303   32847 step_executor.go:389] StepExecutor worker(11): launching oneshot worker
I1115 10:33:44.333402   32847 eventsink.go:59] RegisterResourceOutputs RPC finished: urn=urn:pulumi:my-stack-dev::my-stack::AwsGcpIdentityPoolBinding::awsbindings; err: null, resp: 
I1115 10:33:44.536002   32847 state.go:304] error copying .pulumi/stacks/my-stack-dev.json.gz to .pulumi/stacks/my-stack-dev.json.gz.bak: blob (key ".pulumi/stacks/my-stack-dev.json.gz -> .pulumi/stacks/my-stack-dev.json.gz.bak") (code=NotFound): googleapi: Error 404: No such object: pulumi-bucket/.pulumi/stacks/my-stack-dev.json.gz, notFound
I1115 10:33:44.585400   32847 state.go:310] error deleting source object after rename: .pulumi/stacks/my-stack-dev.json.gz (blob (key ".pulumi/stacks/my-stack-dev.json.gz") (code=NotFound): storage: object doesn't exist) skipping
I1115 10:33:44.893330   32847 eventsink.go:86] eventSink::Error(<{%reset%}>error sending request: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/aws_identity_pool-provider?alt=json</code> was not found on this server.  <ins>That’s all we know.</ins>
<{%reset%}>)

The message about the my-stack-dev.json.gz file being missing seems strange. It's true that the file is not there, but I looked in the bucket where all of the stacks JSON files get stored, and there are no *.gz files for any of the other stacks. I guess the question is: Why is Pulumi concerned that this file is missing for this stack, but has never complained about it for other stacks?

solomonshorser commented 1 year ago

With logging level 9, I got a LOT more information. I was able to find the log entries with the full URL:

I1115 11:02:20.621972   41504 state.go:310] error deleting source object after rename: .pulumi/stacks/my-stack-dev.json.gz (blob (key ".pulumi/stacks/my-stack-dev.json.gz") (code=NotFound): storage: object doesn't exist) skipping
I1115 11:02:20.979627   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>I1115 11:02:20.979578   41576 http.go:107] HTTP Request Begin DELETE https://iam.googleapis.com/aws_identity_pool-provider?alt=json ===================================================
<{%reset%}>)
I1115 11:02:20.979684   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>DELETE /aws_identity_pool-provider?alt=json HTTP/1.1
<{%reset%}>)
I1115 11:02:20.979702   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Host: iam.googleapis.com
<{%reset%}>)
I1115 11:02:20.979716   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>Content-Type: application/json
<{%reset%}>)
I1115 11:02:20.979726   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>User-Agent: Pulumi/3.7.1 (GPN:Pulumi; https://www.pulumi.com) pulumi-google-native/v0.7.0
<{%reset%}>)
I1115 11:02:20.979737   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Request End DELETE https://iam.googleapis.com/aws_identity_pool-provider?alt=json
<{%reset%}>)
I1115 11:02:21.047600   41504 state.go:261] Saved stack my-stack-dev checkpoint to: .pulumi/stacks/my-stack-dev.json (backup=.pulumi/stacks/my-stack-dev.json.bak)
I1115 11:02:21.047672   41504 deployment_executor.go:231] deploymentExecutor.Execute(...): incoming event (nil? false, <nil>)

This is followed by over 1000 lines that are mostly logging (un)marshalling activity. Then there is this:

I1115 11:02:21.456410   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>I1115 11:02:21.456364   41576 http.go:115] HTTP Response Begin DELETE [https://iam.googleapis.com/aws_identity_pool-provider?alt=json ===================================================
<{%reset%}>)
I1115 11:02:21.456447   41504 eventsink.go:78] eventSink::Infoerr(<{%reset%}>HTTP/2.0 404 Not Found
thomas11 commented 1 year ago

Thank you for going through all these logs, @solomonshorser!

When the error occurs, does the provider my_aws_identity_pool-provider exist in GCP or not? Did you check via the UI, for instance? That could tell us if Pulumi is using a wrong URL or whether the delete operation shouldn't be happening at all.

Also, is it possible you made changes to your infra outside of Pulumi? In that case, pulumi refresh can help to get the state back in sync.

solomonshorser commented 1 year ago

The object exists in GCP, according to the UI. I can try to run refresh again, though I think I did that before and it didn't really resolve much.