spacepy / dbprocessing

Automated processing controller for heliophysics data
5 stars 4 forks source link

fast_data: skip deleting files that don't exist on disk #91

Closed jtniehof closed 2 years ago

jtniehof commented 2 years ago

This one-liner fixes an issue with fast_data. It's intended to delete old versions of files but keep their records behind, while setting the exists_on_disk column to indicate it no longer exists.

The problem is with successive runs. It doesn't check this column before trying to delete, so it will delete files it's already deleted.

This PR adds a simple check for this flag and skips the deletion step if the database doesn't think the file exists. If the database thinks it exists and it's not actually there, it still errors--that's a consistentcy problem.

PR Checklist

fast_data is currently untested (its functionality really needs to be wrapped up into main dbp), so there's no test on this.