talkiq / gcloud-aio

(Asyncio OR Threadsafe) Google Cloud Client Library for Python
https://talkiq.github.io/gcloud-aio
264 stars 90 forks source link

Storage.download_to_file and Storage.download_stream ignore CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE #766

Closed SimplySeth closed 3 weeks ago

SimplySeth commented 3 weeks ago

I can list files in the target bucket just fine ... Its when I try using:

Storage.download_to_file and Storage.download_stream that I get a 403.

I can do gcloud cp gs://** just fine using ...

export CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE=~/.path/credentials.json

in my code .. I have ...

def checkCredFile():
    if not os.path.exists(CREDFILE):
        sys.exit("ERROR: Credential file not found: %s" % CREDFILE)
    os.environ["CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE"] = CREDFILE
class ShowRow(ModalScreen):
...

        checkCredFile()
        async with aiohttp.ClientSession() as session:
            client = Storage(session=session)

            blob = await client.download_to_filename(
                LOGBUCKET, name, os.path.join(path, target)
            )

I also tried ...

class ShowRow(ModalScreen):
...
         f = open(os.path.join(path, target), "wb+")
        checkCredFile()
        async with aiohttp.ClientSession() as session:
            client = Storage(session=session)
            blob = await client.download_stream(LOGBUCKET, name, session=session)
            for chunk in blob.iter_lines():
                f.write(chunk)

for comparison .. here is the list code , which works ...

class McApp(App):
...
  async def update_table(self) -> None:
        checkCredFile()
    async with aiohttp.ClientSession() as session:
            self.table.clear()
            self.table.loading = True
            client = Storage(session=session)
            blob = await client.list_objects(LOGBUCKET, params={"prefix": prefix})
            if "items" in blob and len(blob["items"]) > 1:
                items = blob["items"][1:]
                for item in items:
                    self.table.add_row(*(item["name"], item["size"]))
                self.table.loading = False

The credentials I use are in a credentials.json with a service account separate from my personal account

Error:

ClientResponseError: 403, message='Forbidden: <MyNonServicePersonalAccount> does not have storage.objects.get access to the Google Cloud Storage object. Permission
&#39;storage.objects.get&#39; denied on resource (or it may not exist).',
url=URL('https://www.googleapis.com/storage/v1/b/rsg-mcback-offsites-prod-log/o/us7%2Fclstr06-us7-mailchimp%2F202302%2Fmxq82700hb%2Finnobackupex_clstr06-us7-mailchimp-202
30210.log?alt=media')