Create focalplane summary and binned sensor images

LSSTDESC / obs_lsstSim

Configuration and tasks for the simulated LSST camera (lsstSim) for the LSST Data Management Stack

2 stars 0 forks source link

Create focalplane summary and binned sensor images #1

Closed SimonKrughoff closed 6 years ago

kadrlica commented 6 years ago

Instructions for running at NERSC:

source $SCRATCH/../desc/lsstsw/setup.sh
source $SCRATCH/../desc/lsstsw/stack/loadLSST.bash
git clone https://github.com/lsstdesc/obs_lsstSim # clone the appropriate repo
cd obs_lsstSim
git checkout u/krughoff/fp_overview 
setup -k -r . 
scons # build repo
cd $SCRATCH
mkdir input
echo 'lsst.obs.lsstSim.LsstSimMapper' > input/_mapper # make repository
ingestSimImages.py input /global/projecta/projectdirs/lsst/production/DC2/DC2-phoSim-3_WFD-r/output/000000/*.fits.gz # ingest images into the repository
makeFpSummary.py input --id --output output # make the summary and binned file.

kadrlica commented 6 years ago

I've tracked down to references on FITS compression: https://community.lsst.org/t/fits-tile-compression-support/2233 https://community.lsst.org/t/lossless-fits-compression-enabled/2410

The second states that lossless compression is activated by default. The first suggests that compression could be turned off with something like:

binned_sensor_fits_halves:
    recipe: noCompression
    ...

This is based on writeRecipes.yaml.

jchiang87 commented 6 years ago

The storage model we have for the Stack outputs at NERSC assumes, at a minimum, the default compression currently implemented. Is there no way to handle the compressed data on the image-viewer side?

kadrlica commented 6 years ago

Ok, I've made this change to our policy config file and confirmed that it removes compression. Change is committed and pushed.

kadrlica commented 6 years ago

@jchiang87 is storage an issue? These images are downsampled by a factor 16 and we were thinking about only generating them for < 10% of DC2.

The image viewer handles RICE compressed images, so we could potentially add a new recipe to writeRecipes.yaml. (Note that the lossyBasic recipe doesn't work because it gzips the mask plane.) For DES we used globally fpacked images, but I think tile compression should be ok too.

jchiang87 commented 6 years ago

It wasn't clear where you were intending to turn off the compression. From your initial comment it sounded like all compression for all data products coming from the Stack was going to be turned off. If it's just the derived down-sampled images, then indeed I don't care.

kadrlica commented 6 years ago

My intention was just the downsampled images. @SimonKrughoff should confirm.

heather999 commented 6 years ago

@kadrlica is this PR ready to merge? The DM-DC2 folks are interested in having one consistent version of obs_lsstSim, and it seems we are interested in these particular updates for use in the "mini-DRP".

kadrlica commented 6 years ago

@heather999 I think @SimonKrughoff and I decided that these changes didn't require another review, but I'd be happy to perform one if it was deemed necessary.