GCS Token Credentials support

dgaloop commented 3 months ago

🚀 🚀 Pull Request

Impact

[ ] Bug fix (non-breaking change which fixes expected existing functionality)
[ ] Enhancement/New feature (adds functionality without impacting existing logic)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description

The PR aims to adapt deeplake to support Downscroped GCS Credentials, both for Federated and Service JSON (master/permanent) credentials. Additionally, it implements generation of Presigned Urls for streaming of blobs, for the case of linked tensors, with data in GCS attached via corresponding ManagedCredentials

Things to be aware of

Backend is going to downscrope the GCS Service account credentials is possible, which will lead to usage of gcs_oauth_token key, however it will return the whole JSON as previously. Therefore we should still ensure backwards compatilibility in subsequent releases. Additionally, GCS being LinkedCredentials for linked credentials will also avoid downscoping leading to previous response structure.

Things to worry about

Additional Context

Linked Videos in GCS are the only case where the client side, and whatever credentials it gets from backend are not enough to generate presigned urls for blobs. Therefore, a third usecase for GCSStorageProvider emerges - that is generation of presigned url for a given blob, given a ManagedCredentials key.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is `60.81081%` with `29 lines` in your changes missing coverage. Please review.	Files	Patch %
deeplake/core/storage/gcs.py	60.71%	22 Missing :warning:
deeplake/client/client.py	20.00%	4 Missing :warning:
deeplake/core/link_creds.py	50.00%	3 Missing :warning: