Closed Patrickliuu closed 4 months ago
@Patrickliuu I think we might be able to just run the following command on the LightningAI Ubuntu machine: https://cloud.google.com/storage/docs/downloading-objects#downloading-an-object
gcloud storage cp gs://BUCKET_NAME/OBJECT_NAME SAVE_TO_LOCATION
From that same page we can also see the python Code for downloading from Google Cloud Buckets:
from google.cloud import storage
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
# The ID of your GCS bucket
# bucket_name = "your-bucket-name"
# The ID of your GCS object
# source_blob_name = "storage-object-name"
# The path to which the file should be downloaded
# destination_file_name = "local/path/to/file"
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
# Construct a client side representation of a blob.
# Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve
# any content from Google Cloud Storage. As we don't need additional data,
# using `Bucket.blob` is preferred here.
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print(
"Downloaded storage object {} from bucket {} to local file {}.".format(
source_blob_name, bucket_name, destination_file_name
)
)
copy credidentials so the bucket can be downloaded in lightning AI the following commands have to be done in the terminal:
mkdir .config/gcloud
cp gc.json /teamspace/studios/this_studio/.config/gcloud/application_default_credentials.json
gcloud auth application-default login lightning-ci@algorithmic-quartet.iam.gserviceaccount.com
Find out, how the GCS can be used in lightningAI, a possible approach could include: https://github.com/Lightning-AI/litdata, sadly https://lightning.ai/docs/app/stable/workflows/mount_cloud_object_store.html states, that google cloud buckets aren't supported now.