search

sanger / crawler

Store sample data from Lighthouse labs
MIT License
4 stars 3 forks source link

BUG - Make a backup and clear old collections from Mongo prod #95

Open emrojo opened 3 years ago

emrojo commented 3 years ago

Describe the bug Make a backup from old collections and remove them from the database.

RT Ticket Number If applicable

To Reproduce Steps to reproduce the behaviour:

  1. Go to page '...'
  2. Click on button '...'
  3. See error

Expected behaviour What was expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

neilsycamore commented 3 years ago

Which collections are going to be removed?

andrewsparkes commented 3 years ago

@neilsycamore We have adjusted the process so the 'samples' collection is persisted from day to day rather than creating a new one. Katy has created a migration that takes the timestamps from those old collection sample rows and puts them in the persisted samples collection, so we know when the sample rows were originally created. So once that is done we will be able to remove all the old sample collections (as we're out of space and they wouldn't be needed any more). The DBAs back up the database anyway. Does that cause you any issues??

neilsycamore commented 3 years ago

I have a collection that I use db.createView("ditinctNegPlateBarcode3", "samples", [{ $match : { Result : "Negative" } }]) Can I keep this? It's actually misnamed but I created it way back with Phillip and don't now how to rename it but it's a list of -ve samples NOT grouped by plate barcode. I need this to generate a SSR report which is run every Monday. Hoping this will be surpassed eventually by ongoing reporting.. but not yet

andrewsparkes commented 3 years ago

@neilsycamore Think that should be fine as the 'samples' collection is still called 'samples'. We've added some extra columns but not renamed any existing ones. What used to happen is each day the samples collection would get renamed with the date, and a new, latest version 'samples' would be created with everything in it from before plus content from any new files. It's only those old backups we are proposing to remove. It's possible you might get a slightly lower number than before, there was some insertion of incomplete data that shouldn't have been happening before, and some duplications, which we are filtering out and reporting as errors now.