search

desihub / fastspecfit

Fast spectral synthesis and emission-line fitting of DESI spectra.
https://fastspecfit.readthedocs.org
BSD 3-Clause "New" or "Revised" License
13 stars 2 forks source link

consider splitting up the merged catalogs generated with each release #165

Open moustakas opened 7 months ago

moustakas commented 7 months ago

The merged catalogs we generate with each release are getting unwieldy for the main survey. E.g., 

% du /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main*
68M /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-backup.fits
29G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-bright.fits
45G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-dark.fits
73G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main.fits

With ~18M rows (https://fastspecfit.readthedocs.io/en/latest/iron.html#summary-statistics), the fastspec-*-main.fits file is probably going to become too large to be useful, especially in Y3 and Y5.

So we need to think about either splitting this up into small-nside healpixels (like we do for the lsdr9 VAC of potential targets; see here) or maybe even stop producing this file and just provide wrapper code.

Other suggestions are welcome.