failures when running skyCorrection

[johannct]

Example :

CameraMapper INFO: Loading calib registry from /sps/lsst/dataproducts/desc/DC2/Run2.1i/w_2019_17/CALIB/calibRegistry.sqlite3
skyCorr.maskObjects.detection INFO: Detected 3547 positive peaks in 2256 footprints to 2.5 sigma
skyCorr.maskObjects.detection INFO: Detected 4303 positive peaks in 2557 footprints to 2.5 sigma
skyCorr.maskObjects.detection INFO: Detected 4450 positive peaks in 2602 footprints to 2.5 sigma
skyCorr INFO: Background 0: 12260792 pixels
skyCorr INFO: ccwsge0565:63562: Finished processing {'visit': 479477, 'filter': 'r', 'raftName': 'R22', 'detectorName': 'S00', 'detector': 90}
skyCorr FATAL: Failed on dataId={'visit': 479477, 'filter': 'r', 'raftName': 'R22', 'detectorName': 'S00', 'detector': 90}: InvalidParameterError: 
  File "src/math/LeastSquares.cc", line 421, in void lsst::afw::math::LeastSquares::_factor(bool)
    Number of columns of design matrix (1) must be smaller than number of data points (0) {0}
lsst::pex::exceptions::InvalidParameterError: 'Number of columns of design matrix (1) must be smaller than number of data points (0)'

skyCorr INFO: ccwsge0565:63562: Start processing {'visit': 479477, 'filter': 'r', 'raftName': 'R22', 'detectorName': 'S01', 'detector': 91}
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
    result = self.runTask(task, dataRef, kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
    return task.runDataRef(dataRef, **kwargs)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/17.0.1-3-g9730928+12/python/lsst/pipe/drivers/skyCorrection.py", line 140, in runDataRef
    scale = self.sky.solveScales(measScales)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/17.0.1-3-g9730928+12/python/lsst/pipe/drivers/background.py", line 345, in solveScales
    return solve(mask)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_drivers/17.0.1-3-g9730928+12/python/lsst/pipe/drivers/background.py", line 333, in solve
    afwMath.LeastSquares.DIRECT_SVD).getSolution()
lsst.pex.exceptions.wrappers.InvalidParameterError: 
  File "src/math/LeastSquares.cc", line 421, in void lsst::afw::math::LeastSquares::_factor(bool)
    Number of columns of design matrix (1) must be smaller than number of data points (0) {0}
lsst::pex::exceptions::InvalidParameterError: 'Number of columns of design matrix (1) must be smaller than number of data points (0)'

[johannct]

Beyond the failures aesign matrix (1) must be smaller than number of data points (0)' bove (I lost track of the current situation so I need to check the dr1b-v1 logs again), this issue should capture the future decision as to whether we run skyCorrection or not eventually. In https://lsstc.slack.com/archives/C978LTJGN/p1561041974154300 the parameters set for Subaru in order to fix issues reported with skyCorrection there is recalled :

config.detection.doTempWideBackground =   True
config.detection.tempWideBackground.binSize = 128
config.detection.tempWideBackground.useApprox = False
config.detection.reEstimateBackground = True
config.detection.background.binSize = 128
config.detection.background.useApprox = False

and the verbatim from Jim :

The relationship between those options and processCoadd is a red herring, but they may be of interest to obs_lsst. This gets back to the discussion about whether to do a large-scale or small-scale background subtraction and whether to run SkyCorrection: if you recall, I had warned during that discussion that HSC was unhappy with the impacts of large-scale backgrounds on the final catalogs, and we were leaning towards following up the large-scale SkyCorrection subtraction with a small-scale subtraction on the coadds, so the catalog we produce could have the benefits of small-scale subtraction while those interested in very extended objects could still re-insert the small-scale background model in order to make their own catalogs. These configs do exactly that, and were added very recently for the ongoing internal HSC DR. I should also note that these yield a very local background subtraction on the coadds, regarded by at least @rhl as probably too small.

[johannct]

This failure still present in 105 sensors in y3. See https://github.com/LSSTDESC/ImageProcessingPipelines/issues/111#issuecomment-504904524 . This is a small number, but it would be nice to know if something can be done, or if it is expected and entirely ok that skyCorrection fails sometimes. So do we need more forensic study @rearmstr ?