search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

re-factor delete_from_disk_and_database #326

Open whohensee opened 4 months ago

whohensee commented 4 months ago

the FileOnDiskMixin.delete_from_disk_and_database method seems to cause regular issues, and the current implementation (delete from database then disk then archive) could cause some issues with the remove_downstreams flag that can be passed, which generally queries the database to find the objects to remove.

A more robust refactor seems worthwhile. My current idea is to implement a new get_downstreams_recursive function which will get all the downstreams and downstreams of downstreams of an object and return them as a list, then carefully go about deleting all the objects on that list in a proper order. This should solve the issue of querying the database after information has been removed, and will hopefully be more clear as to what is occurring in what order.

rknop commented 4 months ago

I actually did this as part of the conductor PR -- I had to do it towards the end to get rid of some problems with leftover files at the end.