search

ExposuresProvider / cam-pipeline

Data loading pipeline for CAM database
https://exposuresprovider.github.io/cam-pipeline/
MIT License
2 stars 4 forks source link

Rewrite Kubernetes files #124

Open gaurav opened 7 months ago

gaurav commented 7 months ago

The current Kubernetes files are set up for the previous model of cam-pipeline building (where a Kubernetes Job generates it at regular intervals). We seem to have moved onto a model of generating it on-the-fly as needed, so we need far less disk space and we don't need a Kubernetes job any more.

gaurav commented 7 months ago

The first thing to do here would be to see how much disk space we actually need to generate kg.tsv, which I can do on the current job I have running.