Changes in the authorization process for Google Cloud may affect force-level1-csd soon

[ernstste]

Google is deprecating the authentication workflow used by standalone tools such as gsutil, which is part of force-level1-csd. From what I understand, logging in with the current authentication method will not be possible from next month on.

Possible workarounds are using service account authentication for gsutil standalone, authenticating via HMAC, or installing gsutil as a part of the Google Cloud CLI and using its authentication workflow.

So far I wasn't able to look into the different options, but it would be preferable to keep using a gsutil standalone solution. I would be happy to hear feedback if anyone has tried the mentioned methods.

[ernstste]

Just gave the service account authentication a shot - seems easy enough. You have to create a service account in the dashboard, create a key file for the account, and then use the gsutil config command with the -e flag.

This will probably be the preferred method. But again - happy to hear feedback by others.

[davidfrantz]

Thanks for the heads-up!

Does this still generate the .boto?

[ernstste]

It does, but there is a slight change: The path to the keyfile is specified in the .boto config file, so the location must be mounted under the same path when running FORCE in docker.

[davidfrantz]

Stupid question: where do I find the dashboard?

[ernstste]

Sorry David, I didn't see your reply earlier. You probably have found it by now, but just in case: https://console.cloud.google.com --> IAM & Admin (menu visible after clicking the three bars left of the Google logo at the top left) --> Service Account

Our daily downloads went through without issues yesterday and today, so I'm wondering how serious Google was about the announced deadline....

[kelewinska]

FYI, this morning I was not able to set up the connection using the old identification protocol: I got an Access blocked: GSUtil's request is invalid. Unless it is only me, Google has fully migrated to the new identification protocol. Sadly, the service account does not work for me either throwing the following error:

Your "OAuth 2.0 Service Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config").
Could not parse JSON keyfile "/home/[username]/.boto" as valid JSON
CommandException: 1 file/object could not be transferred.

Could it be due to the fact that my google account is secured with two steps identification (pw + token)? It seems I am the only one who struggles with getting data from the google bucket.

EDIT the issue was with my key file. The new identification workflow works (for) now like a charm.

[davidfrantz]

Did they actually change anything? I did not find time and never migrated... But it still works for me

[ernstste]

Good question. It seems like @erfea ran into issues with the old method. I switched to a service account and haven't checked with the old authorization method since.

[kelewinska]

Since January I needed to reconfigure the connection each time i wanted to pull data. later it became very more annoying as pull tasks were interrupted. Since I switched to the new method it works flawlessly.