The ingest code was failing to ingest single new files in Repro. It looks like that was due to a bug in identifying new files that had been downloaded. I also hit a db locking issue that was improved by reorganizing. And while at it, I updated the pieces needed to make a new archive (of the list of files released.. the "cda" h5 piece) to work more happily with shiny (removed six, updated some urllib pieces). I was not rigorous distinguishing reproV from shiny updates here as I have just been working in shiny.
Testing
[x] Passes mica archive l0 unit tests on Linux (a non-related L1 test looks to be failing now)
[x] Functional testing - this worked in a test archive to ingest the files that were failing repeatedly with the flight code on the flight archive. At this point I've also run the test code on the flight archive to just get the files in there.
Description
Fix some L0 ingest hiccups.
The ingest code was failing to ingest single new files in Repro. It looks like that was due to a bug in identifying new files that had been downloaded. I also hit a db locking issue that was improved by reorganizing. And while at it, I updated the pieces needed to make a new archive (of the list of files released.. the "cda" h5 piece) to work more happily with shiny (removed six, updated some urllib pieces). I was not rigorous distinguishing reproV from shiny updates here as I have just been working in shiny.
Testing