simonrw / cratestats.io

1 stars 0 forks source link

Managing database updates #2

Open simonrw opened 4 years ago

simonrw commented 4 years ago

We are currently relying on the database dumps from crates.io. These are large .sql and .json files that are ingested into the postgres database. Downloading these files and updating the database is a slow operation (~20 minutes). We do not want the site to be unavailable at this time, so how can we update the database whilst still serving production traffic on the old data.

Some approaches are: