#13 make fp summary.py broken

[SimonKrughoff]

I believe these two commits fix the focalplane summary issue.

[heather999]

Tested using w_2018_18 + this patched version of LSSTDESC/obs_lsstSim. makeFpSummary ran without complaint on Run1.2p v12449-fy, producing:

Also trying to run w_2018_14 + the pre-fix version of LSSTDESC/obs_lsstSim, but I have been receiving errors.. which I'm hoping is due to the late hour

focalplaneSummary FATAL: Failed on dataId={'visit': 12449, 'filter': 'y'}: TypeError: 'NoneType' object is not subscriptable                                                                
Traceback (most recent call last):                                                            
  File "/global/common/software/lsst/cori-haswell-gcc/stack/lsstsw/weekly/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/15.0-6-gfa9b38f+5/python/lsst/pipe/base/cmdLineTask.py", line 392, in __call__                                                                                
    result = task.run(dataRef, **kwargs)                                                      
  File "/global/cscratch1/sd/heatherk/makeFp/fix/apr/obs_lsstSim/bin/makeFpSummary.py", line 56, in run                                                                                     
    im = cgu.showCamera(butler.get('camera'), imageSource=sbi, binSize=self.config.binSize)   
  File "/global/common/software/lsst/cori-haswell-gcc/stack/lsstsw/weekly/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/15.0-30-g9378914ca/python/lsst/afw/cameraGeom/utils.py", line 991, in showCamera                                                                                    
    **kwargs)                                                                                 
  File "/global/common/software/lsst/cori-haswell-gcc/stack/lsstsw/weekly/stack/miniconda3-4.3.21-10a4fa6/Linux64/afw/15.0-30-g9378914ca/python/lsst/afw/cameraGeom/utils.py", line 913, in makeImageFromCamera                                                                           
    im = imageSource.getCcdImage(det, imageFactory, binSize)[0]                               
TypeError: 'NoneType' object is not subscriptable  

Despite doing setup lsst_distrib -t w_2018_14 before setting up my local obs_lsstSim, it looks like I'm picking up the wrong afw from the weekly builds:

heatherk@cori04:/global/cscratch1/sd/heatherk/makeFp/fix> eups list afw
   14.0-54-g2e4184dee   w_2018_10
   14.0-58-g91bfccb15   w_2018_11
   14.0-60-ga423a1d65   w_2018_12
   14.0-63-g921e36d6a   w_2018_13
   15.0-16-g953e39cab   w_2018_14
   15.0-17-g0b88e3da8+6         w_2018_15
   15.0-21-g91b8abf62   w_2018_16
   15.0-23-g309a1dfe0   w_2018_17
   15.0-24-g02ed2a30c+1         w_2018_18
   15.0-25-gacd707fca   w_2018_19
   15.0-26-g519dd82b0+1         w_2018_20
   15.0-28-g9c8871241   w_2018_21
   15.0-30-g9378914ca   w_2018_22 current setup
   15.0-33-g5c41f601f   w_2018_23

I will look again tomorrow.

[heather999]

Found my brain.. (I neglected to setup with an additional -t w_2018_14 parameter, this is using the weekly builds at NERSC) Ran makeFpSummary using w_2018_14 and the image is a little different but most of the features look similar... the contrast looks different.

[heather999]

Now that Cori is back up - just to make sure.. I reran makeFpSummary using w_2018_14 + the older version of obs_lsstSim starting from ingest, ingestRef cats, processEimage, and finally makeFpSummay. It doesn't seem to have changed the output much, if at all. Is this good enough or do we have to look any deeper [@SimonKrughoff?]

[cwwalter]

I understand (at some level) why we get the ringing on the edges here. But is it concerning that the pattern doesn't look continuous and doesn't always have the same frequency? Is the background fit done per sensor or per focal plane? Or perhaps this some sort of display artifact?

[jchiang87]

I think addressing the ringing in the individual sensors should be discussed and handled in a separate issue from this.

For verifying that the focal plane reconstruction is correct, I would recommend running with --dstype eimage instead of --dstype calexp so that we can use the gradient in the sky background to see more easily if any sensors are out-of-place. I'd also use a visit that has all of the sensors in the focal plane simulated, e.g., using the data in /global/projecta/projectdirs/lsst/production/DC2/DC2-R1-2p-WFD-r/output/000094.

[SimonKrughoff]

@heather999 looks good to me.

@cwwalter the fit is per sensor, so it makes sense that the patters are not necessarily contiguous on the large scale. I think the ringing is dominated by vignetting at the edge, so it makes more sense for there to be larger scale correlations in the ringing pattern at the edge.

[heather999]

I finally finished off @jchiang87's request to rerun on a full visit (/global/projecta/projectdirs/lsst/production/DC2/DC2-R1-2p-WFD-r/output/000094) using eimage rather than calexp. Attached are two very identical looking png files. I know Simon would like this merged and I'll go ahead and do that.