Add requester pays to GCSFilesystem initialization

frodre commented 3 years ago

This PR fixes the gcsfs filesystem initialization after flipping the "requester pays" switch on the gs://vcm-fv3config bucket. The adjustment includes default project ID for the active service account to be used for billing.

From the GCSFS API page:

requester_pays : bool, or str default False

    Whether to use requester-pays requests. This will include your project ID project
    in requests as the userPorject, and you’ll be billed for accessing data from requester-pays
    buckets. Optionally, pass a project-id here as a string to use that as the userProject.

Based on local tests, requester_pays=True should be fine for all of our buckets:

In [1]: import fsspec

In [2]: fsspec.filesystem("gs").exists("gs://vcm-fv3config")
Out[2]: False

In [3]: fsspec.filesystem("gs", requester_pays=True).exists("gs://vcm-fv3config")
Out[3]: True

In [4]: fsspec.filesystem("gs", requester_pays=True).exists("gs://vcm-ml-scratch")
Out[4]: True

In [7]: fsspec.filesystem("gs", requester_pays=True).exists("gs://vcm-ml-public")
Out[7]: True

frodre commented 3 years ago

Slightly confused about daint indicating failure, but I haven't tested with "requester pays" flipped on yet.