zaneselvans
commented
2 years ago It turns out adding {"token": "anon"} to the storage_options and having the bucket objects be readable the the bucket itself be listable was enough.
Closed zaneselvans closed 2 years ago
zaneselvans
commented
2 years ago It turns out adding {"token": "anon"} to the storage_options and having the bucket objects be readable the the bucket itself be listable was enough.
Right now none of the truly public data access methods seems to be functional, which defeats most of the purpose of publishing the data catalog. Several issues have come up:
https://storage.googleapis.comis catastrophically slow and tries to read the entire monolithic parquet file into memory rather than filtering it first, for bothread_parquet()andintakehttps://storage.googleapis.comwithread_parquet()doesn't work easily, since you need to actually list all of the files specifically. No wildcards or directories allowed.https://storage.googleapis.comwithintakeusing a*.parquetwildcard fails with a403 Forbiddenerror because the public user doesn't have permission to list all of the objects in the bucket.gcs://catalyst.coop/intake/testeven though everything in the (pseudo) directory is publicly readable. Do we need to create a separategcs://intake.catalyst.coopbucket that is entirely public, rather than using object-level ACLs? Or is it simply not possible to provide public access overgcs://?intake_parquetcomplains when you give it a directory (filled with parquet files) as itsurlpatheven thoughpandasanddaskare happy to read from a directory. E.g.{{ env(INTAKE_PATH) }}/epacems/does not work. The error says that all paths have to end with.parqor.parquet.Ideally we would be able to provide public access both via
gcs://(which seems to provide much more "filesystem" like access) and overhttps://(which has much better support in generic download tools for the less cloud-literate).Need to understand the intended patterns of usage with public cloud accessible data, and how to make the public resource as functional / convenient as it can be.
May also need to understand better how to limit the risk of a bajillion downloads costing us on data egress fees, which might mean going requester-pays.