search

coiled / dask-bigquery

BSD 3-Clause "New" or "Revised" License
43 stars 13 forks source link

Clean up datasets and tables left behind by CI #71

Open j-bennet opened 1 year ago

j-bennet commented 1 year ago

In this repo, there's a problem with macOS CI jobs. They tend to hang and not finish, and when they do, they leave behind datasets in dask-bigquery-testing project.

This is not urgent, but we need to come up with a plan of how to either avoid this problem, or automate the cleanup.

In BigQuery, it is possible to set expiration for tables within a dataset:

https://cloud.google.com/bigquery/docs/updating-datasets#table-expiration

There doesn't seem to be a similar feature for the dataset. Deleting tables would only solve the problem partially, since each CI job also creates a new temporary dataset.

cc @ncclementi @jrbourbeau

kgantchev commented 7 months ago

Hi, sorry if I'm intruding, but a lot of people seem to be having issues with GitHub's macOS runners. Have you considered giving FlyCI a try? FlyCI offers M2 runners for both public and private repos. FlyCI's runners are more reliable, affordable, and faster than GitHub's. There is chat support via Discord for free tier users and on-demand support for those who purchase the support packages.