Support signurl using compute engine service account

[horgh]

Hi! I noticed in #968 there was functionality added to create signed URLs without a local private key. That's awesome! I'm trying to use this and I could use a little help.

I'm running on a compute instance and I'm able to do things like list the contents of my bucket without setting up any special configuration using the default service account, but trying to create a signed URL results in this message:

$ gsutil signurl -d 10m -u gs://[snip]/[snip].zip
URL     HTTP Method     Expiration      Signed URL
CommandException: Cannot get service account email id for the given credential type.

I'm guessing I need to set up a boto config, though I haven't had to do that for other actions. Is there anything you could suggest?

I'm using gsutil 4.49.

Thank you!

[dilipped]

Hi @horgh, that error indicates that you are not using a service account. Can you check your boto config? Maybe you are using your user account instead of a service account in the boto config. Alternatively, you can impersonate a service account

gsutil -i <service account email id> signurl -d 10m -u gs://[snip]/[snip].zip

More info https://cloud.google.com/storage/docs/gsutil/commands/signurl

[horgh]

Thanks for the response!

I actually have no boto config in use for other gsutil commands that succeed, other than I believe /etc/boto.cfg which is the automatically created one:

# This file is automatically created at boot time by the /usr/lib/python
# 3/dist-packages/google_compute_engine/boto/boto_config.py script.
[snip...]

I figured gsutil is automatically picking up the default compute engine service account. Here's an example (I have no ~/.gcloud or other dotfile directories that appear related to gsutil from what I can tell).

$ rm -rf ~/.gsutil ; gsutil ls gs://[snip]
gs://[snip]/[snip].zip

[dilipped]

You can run gsutil version -l to check the boto config path.

[horgh]

$ gsutil version -l
gsutil version: 4.49
checksum: 89aa91d37decba4899431c0415120312 (OK)
boto version: 2.49.0
python version: 2.7.17 (default, Nov  7 2019, 10:07:09) [GCC 7.4.0]
OS: Linux 5.0.0-1029-gcp
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: False
config path(s): /etc/boto.cfg
gsutil path: /snap/google-cloud-sdk/126/bin/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

[horgh]

$ cat /etc/boto.cfg
# This file is automatically created at boot time by the /usr/lib/python
# 3/dist-packages/google_compute_engine/boto/boto_config.py script. Do
# not edit this file directly. If you need to add items to this file,
# create or edit /etc/boto.cfg.template instead and then re-run
# google_instance_setup.

[GSUtil]
default_project_id = [snip]
default_api_version = 2

[GoogleCompute]
service_account = default

[dilipped]

Interesting! This might need a fix. I guess GCE service accounts were not handled here https://github.com/GoogleCloudPlatform/gsutil/blob/08f696301cbc64f8945080b904517bb6f5fbe099/gslib/gcs_json_api.py#L314 which are one of the creds types https://github.com/GoogleCloudPlatform/gsutil/blob/9af326229303d8636dcb6743b7d8c49f59f50d16/gslib/gcs_json_credentials.py#L375

[horgh]

Oh ok. That sounds promising! I'm not familiar with the code at all :-). I'm happy to test anything if that would help though!

[dilipped]

Edited the title to better describe the exact feature that is needed here.

On digging a little more into this problem, it looks like the fix is not as simple as I thought earlier. GceAssertionCredentials (which is what gets used if using the default setup from a GCE instance) does not provide service_account_email directly which is required for signing the url.

We might have to figure out other ways to access the service account email which would need further exploring the API. A quick look here shows that currently there is no good way to extract that information.

[horgh]

I think it's available via the GCE metadata API, if that's an option. In Go I look it up for SignBlob using this call (passing "default" causes it to return the default one's email).

[horgh]

Hi! Do you think there is any chance this will be supported some time soon?

It has been blocking my team switching to managed private keys in a few spots. We'd love to be able to do this without bespoke code on our side, since so far we've been able to wrap gsutil for what we need!

Thank you!

[eriksw]

I wound up creating a simple tool to generate signed urls that may be useful to those affected by this bug.

retailnext/gcsdropbox creates signed urls using the signBlob api to sign the url using a service account's Google-managed key. The requirements are that you must specify the service account you want to use, and the ambient ADC (Application Default Credentials) must have the iam.serviceAccounts.signBlob permission on the chosen service account (which can be the same service account).

[fjomier]

Hi!

Same problem here with version 4.61 of gsutil. Is there any plan to fix this ? It would be the most secure way for us to sign our GCS url by using our GKE cluster to provide the needed identity, instead of having to handle the service account private key file.

Thanks in advance.

[jdub55]

Also running into this issue. Any updates?

[thomasmaclean]

A quick update after a bit more digging:

It turns out we can fetch the SA email using a single API call (on any given GCE instance):

curl -H "Metadata-Flavor: Google" 'http://metadata/computeMetadata/v1/instance/service-accounts/'

While this is great, we had assumed up until this point that we just needed the SA email and then we'd be able to generate the key. The problem is actually a bit deeper than that, as the actual signing API call happens in oauth2client, which unfortunately does not support signing with GCE service accounts (and never will):

https://github.com/googleapis/oauth2client/issues/471

Seeing what the draft implementation looks like, it seems possible that we could use the IAM API to perform the sign blob operation, although when I did a quick test implementation I kept getting stopped by 400 errors. We'll need to do a bit more digging, but so far I'm not convinced this is feasible.