Add generic GCS download functions and use them in NL server/tools

datacommonsorg / website

Code for the Data Commons website

https://datacommons.org

Apache License 2.0

21 stars 76 forks source link

Closed shifucun closed 2 months ago

shifucun commented 2 months ago

Added 3 generic GCS functions:

download_blob() which downloads a blob(file or folder) given bucket, blob name and local path.
download_blob_by_path() which is a wrapper of download_blob() that takes gcs path
maybe_download() which downloads the blob under 'local_path_prefix' and keeps the gcs path structure, and skip the download if already exists.

Use these functions throughout NL apps and remove unused functions.

shifucun commented 2 months ago

Thanks for the review!