search

moustakas / legacyhalos

Code and papers for a multi-faceted study of the baryonic content of dark matter halos in Legacy Survey imaging.
MIT License
4 stars 7 forks source link

do not write out ccddata and also, maybe, remove pickle files when done #80

Closed moustakas closed 5 years ago

moustakas commented 5 years ago

The biggest data hogs of the pipeline right now are the individual CCD images that went into building the custom coadds and the srcs pickle files. Although I would love to have the CCD images written out, I don't think making an entire copy of all the imaging that went into a stack is justified -- the information is there (in the -ccds.fits file) to grab those images post-facto if necessary.

Similarly, the srcs and fitblobs checkpoint/pickle files are useful for when the processing gets interrupted (e.g., by an expiring queue), but once the final stack is created these files should just get blown away, I think.

Or they can be cleaned up after the fact with a dedicate script -- whatevs.

moustakas commented 5 years ago

Done in #87.

I'm keeping the pickle files for now but the gigantic CCD files are now only optionally written.