Add logic to save Dataframe directly to GCS as CSV and then load the CSV to Bigquery

kevinsunny1996 / E2E-Data-Science-Pipelines

Contains Airflow pipelines to perform ELT and analysis on RAWG API data

1 stars 0 forks source link

Closed kevinsunny1996 closed 6 months ago

kevinsunny1996 commented 11 months ago

[ ] Add a variable specifying the GCS bucket to save RAWG API extracts
[ ] Save the dataframe to the GCS path fetched from the variable using to_csv method

kevinsunny1996 commented 9 months ago

Extract phase - Gathering data from API and saving to GCS has been accomplished under #6

kevinsunny1996 commented 9 months ago

Points todo to complete the Load stage of the datapipeline:

[ ] Create a code to scan the latest 5 files (there are only 5 dataframes being saved as of now) and load them to Bigquery using GCSToBigQueryOperator.
[ ] Move the loaded files to nearline or archive storage to save on costs.