incremental data ingestion

ibis-project / ibis-analytics

Ibis analytics, with Ibis (and more!)

https://ibis-project.github.io/ibis-analytics

MIT License

19 stars 8 forks source link

incremental data ingestion #27

Open lostmygithubaccount opened 1 year ago

lostmygithubaccount commented 1 year ago

currently, all data from all time for PyPI and GitHub is ingested on every CI run. should incrementally update the data (download the backup, check last timestamp(s), only ingest new data)