Open moustakas opened 7 months ago
The merged catalogs we generate with each release are getting unwieldy for the main survey. E.g.,
% du /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main* 68M /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-backup.fits 29G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-bright.fits 45G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main-dark.fits 73G /global/cfs/cdirs/desi/public/dr1/vac/dr1/fastspecfit/iron/v2.0/catalogs/fastspec-iron-main.fits
With ~18M rows (https://fastspecfit.readthedocs.io/en/latest/iron.html#summary-statistics), the fastspec-*-main.fits file is probably going to become too large to be useful, especially in Y3 and Y5.
fastspec-*-main.fits
So we need to think about either splitting this up into small-nside healpixels (like we do for the lsdr9 VAC of potential targets; see here) or maybe even stop producing this file and just provide wrapper code.
lsdr9
Other suggestions are welcome.
The merged catalogs we generate with each release are getting unwieldy for the main survey. E.g.,
With ~18M rows (https://fastspecfit.readthedocs.io/en/latest/iron.html#summary-statistics), the
fastspec-*-main.fits
file is probably going to become too large to be useful, especially in Y3 and Y5.So we need to think about either splitting this up into small-nside healpixels (like we do for the
lsdr9
VAC of potential targets; see here) or maybe even stop producing this file and just provide wrapper code.Other suggestions are welcome.