Interface between GC Storage and lightningAI

[Patrickliuu]

Find out, how the GCS can be used in lightningAI, a possible approach could include: https://github.com/Lightning-AI/litdata, sadly https://lightning.ai/docs/app/stable/workflows/mount_cloud_object_store.html states, that google cloud buckets aren't supported now.

[GeroVanMi]

@Patrickliuu I think we might be able to just run the following command on the LightningAI Ubuntu machine: https://cloud.google.com/storage/docs/downloading-objects#downloading-an-object

gcloud storage cp gs://BUCKET_NAME/OBJECT_NAME SAVE_TO_LOCATION

[GeroVanMi]

From that same page we can also see the python Code for downloading from Google Cloud Buckets:

from google.cloud import storage

def download_blob(bucket_name, source_blob_name, destination_file_name):
    """Downloads a blob from the bucket."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"

    # The ID of your GCS object
    # source_blob_name = "storage-object-name"

    # The path to which the file should be downloaded
    # destination_file_name = "local/path/to/file"

    storage_client = storage.Client()

    bucket = storage_client.bucket(bucket_name)

    # Construct a client side representation of a blob.
    # Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve
    # any content from Google Cloud Storage. As we don't need additional data,
    # using `Bucket.blob` is preferred here.
    blob = bucket.blob(source_blob_name)
    blob.download_to_filename(destination_file_name)

    print(
        "Downloaded storage object {} from bucket {} to local file {}.".format(
            source_blob_name, bucket_name, destination_file_name
        )
    )

[Patrickliuu]

copy credidentials so the bucket can be downloaded in lightning AI the following commands have to be done in the terminal:

mkdir .config/gcloud cp gc.json /teamspace/studios/this_studio/.config/gcloud/application_default_credentials.json
gcloud auth application-default login lightning-ci@algorithmic-quartet.iam.gserviceaccount.com