search

googleapis / nodejs-storage

Node.js client for Google Cloud Storage: unified object storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.
https://cloud.google.com/storage/
Apache License 2.0
891 stars 367 forks source link

"Failure from metadata server" when using GKE and Workload Identity #2346

Closed bradam12 closed 4 months ago

bradam12 commented 8 months ago

We were getting the "Failure from metadata server" when trying to create a signed URL in our existing application. This is an app already running in prod. We're adding an additional feature that requires signed URLs.

I created a barebones test.js file and also used the included sample at https://github.com/googleapis/nodejs-storage/blob/main/samples/generateV4ReadSignedUrl.js as reproducible cases. Versions 2.x, 3.x, 4.x all work properly, and 5.0.0 and up exhibit the error.

Environment details

Steps to reproduce

  1. Run node:16 pod in GKE (we're currently on v1.24.15-gke.1700), with established workload identity, KSA and GSA linked. GSA has Storage Object Admin and Service Account Token Creator.
  2. Exec into pod
  3. Write the sample file from https://github.com/googleapis/nodejs-storage/blob/main/samples/generateV4ReadSignedUrl.js
  4. Install @google-cloud/storage@5
  5. Run node test.js bucketname filename
  6. Receive error: Failure from metadata server.
  7. Repeat for @google-cloud/storage@6 and @google-cloud/storage@7, receive same error
ddelgrosso1 commented 8 months ago

Hi @bradam12 is this only happening within GKE? Has it been successful in other environments for you?

bradam12 commented 8 months ago

Local is fine. I can spin up a GCP VM and try similarly.

On Fri, Oct 27, 2023, 9:33 AM Denis DelGrosso @.***> wrote:

Hi @bradam12 https://github.com/bradam12 is this only happening within GKE? Has it been successful in other environments for you?

— Reply to this email directly, view it on GitHub https://github.com/googleapis/nodejs-storage/issues/2346#issuecomment-1782928357, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZHDEGD3WCNTYGDCOVRM3YBOZZNAVCNFSM6AAAAAA6SBBAW6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBSHEZDQMZVG4 . You are receiving this because you were mentioned.Message ID: @.***>

ddelgrosso1 commented 8 months ago

@bradam12 came across this documentation: https://cloud.google.com/knowledge/kb/unable-to-reach-gke-metadata-server-on-workload-identity-enabled-cluster-000004705 is this inline with your problem?

bradam12 commented 8 months ago

Unfortunately does not apply. I'm able to curl to the metadata server to get a token correctly.

bradam12 commented 4 months ago

I'm no longer working in that env. Closing.