search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

Deleting one object out of a file from archive #207

Closed guynir42 closed 2 months ago

guynir42 commented 4 months ago

Example: we have a list of Cutouts objects, and we saved it to disk and to archive, but then decided we want to delete one (or a few) objects from the list.

Then to delete the local file is easy, just remove the dataset (either HDF5 group or the FITS HDU) and if the file doesn't have any more datasets, delete the file.

BUT, if there's also a file on the archive then we are in trouble because you need to send a request to delete a part of a file, and that is currently not supported.

guynir42 commented 4 months ago

This also changes the MD5sum so it is not going to be trivial to do.

Perhaps what is best is just to overwrite the file on the archive. Then we'd need to update all other Cutouts that the sum has changed...

guynir42 commented 2 months ago

I'm closing this as I am coming around to thinking this is not allowed by the logic of how we manage data right now.