mount helper does not work with federated workload identity

[martinkaberg]

I am not able to get mount helper to work with federated workload identity. Just running gcsfuse works fine on the same system. Outputs below.

mount -t gcsfuse  velen-wp-uploads-germany /tmp/bucket
Calling gcsfuse with arguments: -o rw velen-wp-uploads-germany /tmp/bucket
2022/04/28 13:40:52.056394 Start gcsfuse/v0.41.0 (Go version go1.18.1) for app "" using mount point: /tmp/bucket
2022/04/28 13:40:52.067447 Opening GCS connection...

terminal stalls for a few minutes

then prints this message

2022/04/28 13:43:43.084844 Failed to open connection: GetTokenSource: DefaultTokenSource: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
daemonize.Run: readFromProcess: sub-process: mountWithArgs: getConnWithRetry: GetTokenSource: DefaultTokenSource: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
running gcsfuse: exit status 1

On the same system i am able to mount using the gcsfuse command

gcsfuse velen-wp-uploads-germany /tmp/bucket
2022/04/28 13:39:41.808885 Start gcsfuse/v0.41.0 (Go version go1.18.1) for app "" using mount point: /tmp/bucket
2022/04/28 13:39:41.820575 Opening GCS connection...
2022/04/28 13:39:42.050355 Mounting file system "velen-wp-uploads-germany"...
2022/04/28 13:39:42.050891 File system has been successfully mounted.
root@ip-10-200-1-18:/# ls /tmp/bucket
'Image from iOS (13).gif'

[martinkaberg]

So found out what the issue is GOOGLE_APPLICATION_CREDENTIALS environment variable is not available when mount executes gcsfuse. It would be nice if --key-file option supported workload identity, or if we could have another option for that file.

Now i created a simple work around. I renamed gcsfuse binary to _gcsfuse and wrote bash script called gcsfuse

cat gcsfuse 
#!/bin/bash
export GOOGLE_APPLICATION_CREDENTIALS=/var/run/secrets/tokens/gcp-ksa/google-application-credentials.json
/usr/local/bin/_gcsfuse $@

[avidullu]

Thanks for the helpful feedback here! We'll investigate the feasibility of using workload identity in gcsfuse.

[amoghmishra-sl]

Any fixes around it?

[Tulsishah]

GOOGLE_APPLICATION_CREDENTIALS

Hi @amoghmishra-sl, as I understood, mount helper is running through the root, and the GOOGLE_APPLICATION_CREDENTIALS environment variable is not set in root environment. You can try to set this environment variable in the root mentioned like this and check if it is working or not.

Let me know if it works for you.

Thanks, Tulsi Shah.

[vadlakondaswetha]

@martinkaberg - Can you share details of the machine and how the workload identity is setup.