spacepy / dbprocessing

Automated processing controller for heliophysics data
5 stars 4 forks source link

fast_data: add option to archive files instead of delete. #72

Closed jtniehof closed 3 years ago

jtniehof commented 3 years ago

This requires a little background. The fast_data.py script solves a particular operational problem we have with PSP, where we generate new versions at a far greater rate than we want to keep. This script is set up to delete files that are not the newest version but are newer than a certain date, so we can clean up after the period when data are downlinking and new versions being made on an hourly basis. It's not fully documented and long-term I need to generalize parts of it, get the documentation going, etc.

But in the short term, I needed to archive the files rather than delete. This involves marking as not on disk, and move into another directory not managed by dbprocessing, so the files are around but "dead to dbprocessing".

This also picked up a SpacePy dependency for this particular script.

PR Checklist

Since there are no docs for this now, I didn't have time to write up the entire script and just add in the docs of what changed.

jtniehof commented 3 years ago

Looks like the docker auth (#51) isn't working anymore. Will have to track this down later.

jtniehof commented 3 years ago

Got CircleCI running by making a new Docker user, but now there's an issue where SQLAlchemy has changed the name for postgres, https://stackoverflow.com/questions/62688256/sqlalchemy-exc-nosuchmoduleerror-cant-load-plugin-sqlalchemy-dialectspostgre . I'll have to see how early the new name is available and open another PR.

jtniehof commented 3 years ago

'postgresql' as the dialect name goes back to at least 0.6.