Config `gcp:credentials` not observed

[gunar]

What happened?

Docs say you can set a config named gcp:credentials in order to provide the credentials. However, that is not the case. I'm getting "failed to load application credentials".

Using GOOGLE_CREDENTIALS=credentials.json pulumi preview works but I want to be able to store the credentials as a secret config in the repo.

Expected Behavior

Credentials from the config gcp:credentials should be used.

Steps to reproduce

$ pulumi config set gcp:credentials <credentials.json
$ pulumi preview # fails

Output of pulumi about

CLI
Version 3.53.1 Go Version go1.19.5 Go Compiler gc

Plugins NAME VERSION cloudflare 4.15.0 docker 3.6.1 gcp 6.48.0 nodejs unknown postgresql 3.6.0 random 4.10.0

Host
OS nixos Version 22.11 (Raccoon) Arch x86_64

This project is written in nodejs: executable='/run/current-system/sw/bin/node' version='v16.18.1'

Backend
Name pulumi.com URL https://app.pulumi.com/satoshienergy User satoshienergy Organizations satoshienergy

Dependencies: NAME VERSION @pulumi/docker 3.6.1 @pulumi/gcp 6.48.0 @pulumi/postgresql 3.6.0 @pulumi/random 4.10.0 jsonwebtoken 9.0.0 node-forge 1.3.1 @pulumi/cloudflare 4.15.0 @pulumi/pulumi 3.53.1 @types/jsonwebtoken 9.0.1 @types/node-forge 1.3.1 @types/node 18.11.18

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[squaremo]

I did not manage to recreate this problem. I did this:

1. create a new Pulumi project pulumi new gcp-javascript
2. make sure I'm not logged into Google Cloud: gcloud auth application-default revoke
3. create a service account in the Google Cloud console, and download the JSON key
4. Do pulumi preview and observe that it fails ("failed to load application credentials")
5. Run pulumi config set --secret gcp:credentials <path/to/key.json
6. Do pulumi preview and observe that it works

[squaremo]

pulumi config set gcp:credentials <path/to/key.json also worked (without the --secret -- but I would recommend using --secret)

[squaremo]

How did you obtain the credentials.json? I created a service account, then a key for the service account, and downloaded the private key as JSON.

[gunar]

Can't reproduce your reproduction. Step 6 fails:

 ❯ p preview --refresh
Previewing update (bar)

View Live: https://app.pulumi.com/<REDACTED>/foo/bar/previews/00e060d7-e1f7-4b28-8ec2-f715cfb49e25

     Type                   Name       Plan       Info
 +   pulumi:pulumi:Stack    foo-bar    create
     └─ gcp:storage:Bucket  my-bucket             1 error

Diagnostics:
  gcp:storage:Bucket (my-bucket):
    error: failed to load application credentials.
    To use your default gcloud credentials, run:
        `gcloud auth application-default login`
    See https://www.pulumi.com/registry/packages/gcp/installation-configuration/ for details.

How did you obtain the credentials.json? I created a service account, then a key for the service account, and downloaded the private key as JSON.

Yes, same here.

This is what it looks like:

{
  "type": "service_account",
  "project_id": "<REDACTED>",
  "private_key_id": "<REDACTED>",
  "private_key": "<REDACTED>",
  "client_email": "<REDACTED>",
  "client_id": "<REDACTED>",
  "auth_uri": "<REDACTED>",
  "token_uri": "<REDACTED>",
  "auth_provider_x509_cert_url": "<REDACTED>",
  "client_x509_cert_url": "<REDACTED>"
}

[squaremo]

Wow, how confusing! (Another confusing thing: I deleted the key in the GCP console, but pulumi preview still works ..?) I'm asking around to see if anyone else has seen this.

[rquitales]

So 2 things:

1. The error message we see here (error: failed to load application credentials....) is actually quite generic upon reviewing our code and removes a lot of debugging info. This is because we load and validate the config whenever the provider is used, but always assumes the error returned from validation is due to unset credentials. Code: https://github.com/pulumi/pulumi-gcp/blob/d8be576c1640b48cbdde9d842cd3e5db8d0297b7/provider/resources.go#L239-L245

We should improve the error messaging to make debug easier since there could be a wide range of issues that causes validation to fail in config.LoadAndValidate when you inspect that function further. What's odd however is that using the credentials supplied as a path within the ENV var works, but supplying it as a content within the Pulumi config seems to fail. This leads me to suspect that it might be some sort of serialization or encoding issue as a first guess as to what might be happening.

@gunar could you try pulumi config set gcp:credentials path_to_credentials.json (note the lack of <). This will dump the path to your credentials file instead of its contents to the Pulumi config and see if this works. Example Pulumi config for reference:

config:
  gcp:credentials: /home/user/credentials.json
  gcp:project: fake-project

2. @squaremo pulumi preview still works even after disabling/deleting the key in GCP as preview is compared against the existing state. So it would be expected that pulumi preview should work despite the key being deleted, but running any pulumi commands that interact with the live state (eg. pulumi up) is expected to fail due to improper auth. It also doesn't look like the validation function I mentioned earlier actually does a roundtrip to ensure that the token is fresh, just that its structure is valid.

[gunar]

@gunar could you try pulumi config set gcp:credentials path_to_credentials.json (note the lack of <). This will dump the path to your credentials file instead of its contents to the Pulumi config and see if this works.

Just tried it but it didn't work. Same error.

[squaremo]

pulumi preview still works even after disabling/deleting the key in GCP as preview is compared against the existing state. So it would be expected that pulumi preview should work despite the key being deleted

pulumi preview --refresh also worked, IIRC. Perhaps this can be explained by there being no live system -- I never got as far as running pulumi up -- so there's nothing to check the state against.

[squaremo]

@gunar could you try pulumi config set gcp:credentials path_to_credentials.json (note the lack of <). This will dump the path to your credentials file instead of its contents to the Pulumi config and see if this works.

Just tried it but it didn't work. Same error.

The other thing that occurred to me is that there's some oddity in your environment. Is there any chance you can try the repro from a fresh environment -- maybe within a Docker container? This is going far out of your way, I know -- thank you for persevering.

[gunar]

maybe within a Docker container?

Here are the results:

	gcp-javascript	my project
container	:white_check_mark:	:x:
host	:white_check_mark:*	:x:

* Not sure what changed but it's working now.

Not sure what could be wrong with my ~development environment~ project. I have even deleted ~/.config/gcloud just in case but no dice.

[gunar]

Actually, after deleting ~/.config/gcloud the first project I've created out of the template gcp-javascript stopped working too. But creating a new project from the template worked again.

[gunar]

Sorry for all the confusion. After doing the steps below, I can consistently get gcp-javascript to work but my project won't.

• deleted ~/.pulumi/plugins
• deleted ~/.config/gcloud`
• npm install
• pulumi config set gcp:credentials --secret <cred.json
• pulumi preview --refresh (works)

I don't see any difference between my project and these created by the template, so I'm lost here. I can't provide repro instructions as the code is proprietary.

NB: I'm using the same cred.json file though.

[squaremo]

I don't see any difference between my project and these created by the template, so I'm lost here. I can't provide repro instructions as the code is proprietary.

Is there a way you can remove the proprietary bits; or otherwise boil it down to a still failing, but shareable form? (E.g., comment it all out, add things in until it stops working -- it's low-fi but might be all we can do!)

[gunar]

I think I was able to boil it down to a minimum repro.

It seems setting gcp:credentials doesn't work if the project had previously been deployed with a different authentication method.

Works

1. pulumi new gcp-javascript
2. pulumi config set gcp:credentials --secret<creds.json
3. pulumi preview --refresh

Doesn't work

1. pulumi new gcp-javascript
2. GOOGLE_CREDENTIALS=creds.json pulumi up --refresh --yes (works, deploys)
3. pulumi config set gcp:credentials --secret<creds.json
4. pulumi preview --refresh (doesn't work, fails)

Are you able to repro?

[gunar]

Hey, I see #995 got merged. What do you need from me to moving this issue forward?

[squaremo]

@gunar Sorry for the period of silence, and thanks for the repro. I think it's with us, to try the repro first. If we are stuck, it might help for you to try it with the PR that was merged above -- but it's not in a release yet.

[gunar]

Guess what, I'm hitting this problem again :upside_down_face:

[gunar]

I think I've discovered part of the problem. It seems that --refresh is trying to use the outdated credentials instead of the new ones.

• Works: pulumi preview
• Doesn't work: pulumi preview --refresh

[mrroll]

I'm getting this, too on a project previously authenticated with using the GOOGLE_CREDENTIALS environment variable. Are there any workarounds we can use for now?

[cooervo]

I'm also getting this error with no solutions found online

[VenelinMartinov]

Hey folks, looks like you are hitting https://github.com/pulumi/pulumi/issues/4981. I believe the workaround is to run pulumi up on the provider first, in order to refresh the credentials before running pulumi refresh.

We generally suggest you use the environment variable instead of the config for credentials wherever possible.

If this does not work, I'd really appreciate it if you could include step by step instructions on how to reproduce your issue.

[cooervo]

Hey folks, looks like you are hitting pulumi/pulumi#4981. I believe the workaround is to run pulumi up on the provider first, in order to refresh the credentials before running pulumi refresh.

We generally suggest you use the environment variable instead of the config for credentials wherever possible.

If this does not work, I'd really appreciate it if you could include step by step instructions on how to reproduce your issue.

@VenelinMartinov, that's a valid workaround but the issue is when we need different GOOGLE_CREDENTIALS per stack and per envs it becomes more difficult to manage, it would have been ideal with gcp:credentials.