search

openclimatefix / dagster-dags

Dags for running jobs on Leonardo
2 stars 0 forks source link

Passiv Data local storage #119

Closed devsjc closed 1 month ago

devsjc commented 2 months ago

Currently, although passiv is a "cloud archive", the parquet files it produces are filling up /home/dagster (~70GB at the moment). This is due to the path defined in https://github.com/openclimatefix/dagster-dags/blob/4ba97e7cf6541d8bd219c9fcc73b6333cad82937/cloud_archives/pv/passiv/passiv_year.py#L52 defaulting to the dagster root directory.

There are two possible solutions to this:

- Delete the parquet file after sucessful upload
- Change the path to store the data on the hard disks
peterdudfield commented 2 months ago

Ah thanks, I'll add a delete. Feel free to delete on local storage

peterdudfield commented 1 month ago

I think this is ready to deploy now