search

voxel51 / eta

ETA: Extensible Toolkit for Analytics
https://voxel51.com
Apache License 2.0
29 stars 13 forks source link

Generate signing credentials via IDTokenCredentials when using default Google credentials #595

Closed brimoor closed 1 year ago

brimoor commented 1 year ago

The following should work on a GCE instance with default credentials that have:

import io
import requests
from PIL import Image

import eta.core.storage as etas

cloud_path = "gs://voxel51-test/quickstart/000880.jpg"

client = etas.GoogleCloudStorageClient()

# Signing should work
url = client.generate_signed_url(cloud_path)
r = requests.get(url)
img = Image.open(io.BytesIO(r.content))
assert r.status_code == 200
assert img.size == (640, 480)

# Direct downloads should also work
client.download(cloud_path, "000880.jpg")
findtopher commented 1 year ago

This doesn't appear to be working - topher.dev.fiftyone.ai

The service account managing the GKE cluster has been granted Service Account Token Creator permissions but we get the following error on the fiftyone-app pods:

❯ stern fiftyone-app --since 24h --no-follow --timestamps
+ fiftyone-app-657d88b8bf-f6rfh › fiftyone-app
+ fiftyone-app-657d88b8bf-dvk57 › fiftyone-app
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T15:40:05.352806257-04:00 [2023-05-09 19:40:05 +0000] [23] [INFO] Running on http://0.0.0.0:5151 (CTRL + C to quit)
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:54.363474608-04:00 Migrating dataset 'quickstart-groups' to v0.20.1
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935493935-04:00 ERROR:root:403 GET https://storage.googleapis.com/storage/v1/b/voxel51-test?projection=noAcl&prettyPrint=false: Caller does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935534935-04:00 Traceback (most recent call last):
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935539973-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/server/decorators.py", line 34, in wrapper
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935543269-04:00     response = await func(endpoint, request, data, *args)
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935546099-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/server/routes/samples.py", line 29, in post
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935549425-04:00     results = await paginate_samples(
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935553330-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/server/samples.py", line 131, in paginate_samples
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935563782-04:00     nodes = await asyncio.gather(
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935568549-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/server/samples.py", line 178, in _create_sample_item
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935572574-04:00     metadata = await fosm.get_metadata(
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935577336-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/server/metadata.py", line 82, in get_metadata
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935581640-04:00     filepath_result, filepath_source, urls = await _create_media_urls(
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935585726-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/server/metadata.py", line 546, in _create_media_urls
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935590162-04:00     url = foc.media_cache.get_url(path, method="GET", hours=24)
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935594800-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/core/cache.py", line 338, in get_url
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935598926-04:00     return _get_url(client, remote_path, method=method, hours=hours)
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935603103-04:00   File "/opt/fiftyone-teams-app/lib/python3.10/site-packages/fiftyone/core/cache.py", line 909, in _get_url
fiftyone-app-657d88b8bf-dvk57 fiftyone-app 2023-05-09T16:09:56.935607370-04:00     return client.generate_signed_url(remote_path, **kwargs)

Using a Service Account Key as the same principle everything seems to work just fine.