LMFDB / lmfdb-inventory

inventory of the lmfdb database
3 stars 14 forks source link

Copy Inventory database & collections to cloud #77

Closed JohnCremona closed 6 years ago

JohnCremona commented 6 years ago

Before we next push to prod we need to copy the new database "inventory" and its collections (DB_ids, collection_ids, fields_auto, fiuelds_human, indexes, records, rollback -- or possibly only some of these) -- to the cloud. I will check to see which collections are needed for www.lmfdb.org/inventory to work in the read-only setting there. After that we should add these collections to the list of those which are automatically updated frequently, as we currently do for knowls. I expect that @edgarcosta knows where that is configured. I'll comment back here when I know exactly which collections will be needed on the cloud.

JohnCremona commented 6 years ago

The answer is

the first 6 definitely need copying. Those are the actual inventory content.

The rollback doesn't need copying, as it just stores recent changes in case of issues, so is only needed when editing inventory. For future reference this is a Mongo capped collection, so it doesn't grow out of control, which does change some of its behaviour.

If you could instead create an empty collection on the cloud db named "rollback" please, because the way I have it currently coded it expects it to be there. I'll fix that in the round of changes we'll likely have once anybody tries this out.

and I just copied all 7 for simplicity. Someone could delete all the content from the rollback collection (but leave the collection in place). I used the usual script update-cloud-db.sh so had 7 pairs of warnings about things not existing and that does seem to have prevented the copies being made since http://www.lmfdb.org/api/ shows nothing there. @edgarcosta would it be enough to comment out line 61 of the update script, or do we have to create the inventory database manually first?

JohnCremona commented 6 years ago

I also added update_inventory scripts to the lmfdb-gce repository and would be grateful for someone to look and see if it looks good.

edgarcosta commented 6 years ago

I will look into it

edgarcosta commented 6 years ago

I have fixed the scripts, uploaded the inventory, and cleaned up the leftovers.

How often do we want to automatically update the inventory?

edgarcosta commented 6 years ago

I set it up for every 20 mins.

JohnCremona commented 6 years ago

Thanks, that sounds about right. We can review it later, when the system has been running for a while.