Get files from GCS in utils.gcs_to_gbq

dataverbinders / statline-bq

Library to fetch CBS open datasets into parquet and optionally load into Google Cloud Storage and BigQuery

MIT License

0 stars 0 forks source link

Get files from GCS in utils.gcs_to_gbq #46

Open galamit86 opened 3 years ago

galamit86 commented 3 years ago

Currently, when making the link from gcs to bq, the file names used are taken from a previous steps, and represent the local files that should have been upload the GCS.

It makes more sense to download the actual file names and other data from GCS instead.

Further comments in the code.