Run QA analysis scripts on DC1 output

[jchiang87]

The procedure is described by Lauren here: https://jira.lsstcorp.org/browse/DM-11452

[jchiang87]

@jamesp-epcc @heather999 Could you work together on this please? We can use the new v14_0 installation for this.

[heather999]

The v14 install is in progress, but since then we've learned that the current sims tag is not compatible with v14. Scott has suggested w.2017.39 (see the sims channel). Ok to go with that weekly plus sims or does this exercise really only need lsst_distrib?

The Slack messages on desc-sim mentioned that we might need a copy of the data copied to NERSC or is it already in your scratch space @jchiang87 ?

[jchiang87]

For this exercise, we don't actually need the sims distribution. The data for this would be the imsim-dithered output that Tony copied to the production area:

/global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered

Based on the instructions at the JIRA issue, I would set ROOTDIR to that directory, and RERUNDIR to some local directory in the scratch area.

[heather999]

I tried to build the obs_lsstSim branch against v14, but I ran into an error - which looks like a compatibility issue (due to the recent camera geometry updates?):

python bin.src/makeLsstCameraRepository.py --clobber
Traceback (most recent call last):
  File "bin.src/makeLsstCameraRepository.py", line 40, in <module>
    from lsst.afw.cameraGeom import DetectorConfig, CameraConfig, \
ImportError: cannot import name 'TransformMapConfig'
scons: *** [description/camera] Error 1
scons: building terminated because of errors.

We should coordinate on the DM JIRA to get this sorted out, I see this was previously tested using the "shared stack" (at NCSA?) I can't currently log into the DM JIRA area to post a comment, I'll see if I can get that sorted out.. in the meantime, @jchiang87 can you inquire? https://jira.lsstcorp.org/browse/DM-11452

[jchiang87]

@laurenam Can you comment on the error that @heather999 is running into please? Thanks!

[laurenam]

This relates to the implementation of RFC-346. In particular, among many other changes it states there that:

geom.TransformMapConfig and geom.TransformConfig shall be modified to use transformRegistry rather than xyTransformRegistry.

So, yes, you will need to update to the current CameraGeom API. I did this for meas_mosaic (https://github.com/lsst/meas_mosaic/commit/c4743ab86450402eef38e2b4518de0e9613153c4), but was hand-held through it by @r-owen and would suggest asking him for help if you're unsure of anything.

[laurenam]

Oh, wait, the error you're seeing is the update to the new API: https://github.com/lsst/obs_lsstSim/commit/62de841c621b01cb81e114918b8b008b1e1f252f Perhaps v14 is pre-API change, so you may just need to revert that commit to compile against it?

[heather999]

Thanks for the response! Yes, I'll definitely need some further directions. Was there a specific weekly that you initially started from @laurenam or was it v14?

[laurenam]

I'm almost always using the "latest" weekly, so was likely last using w_2017_41 and obs_lsstSim compiled fine against it. It looks like the cameraGeom API change went in on 36 (see https://lsst-web.ncsa.illinois.edu/~swinbank/changelog_weekly.html for a handy changelog).

[laurenam]

Indeed, the cameraGeom update went in after the v14 tag: https://github.com/lsst/obs_lsstSim/compare/14.0...master

[jamesp-epcc]

Hi, sorry for the lateness of my response to this. One of my other projects has been demanding a lot of my time this week, but happy to help out with this.

[jamesp-epcc]

I've had a look at this now, but am also getting the same error when building obs_lsstSim. Please let me know if you work out how to solve this.

[heather999]

@jamesp-epcc I think I've almost got it :) I'm working on this right now and hope to have it set up in a bit.

[heather999]

And now for a long tale.. Long story short, I need some guidance. We have w.2017.41 with Py3 (which includes the camera geometry updates) installed at NERSC in /global/common/software/lsst/cori-haswell-gcc/stack/w_2017_41, owned by the shared desc account. That took a couple tries, there was a transient error when testing meas_base, which disappeared upon retry. Others can use it, and have lsst_distrib set up for them by source /global/common/software/lsst/cori-haswell-gcc/stack/setup_w_2017_41.sh Now under my own user account, I installed the branches of obs_lsstSim and pipe_analysis, following the directions as outlined in the DM JIRA, but when testing visitAnalysis, I encounter an error: visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun $SCRATCH/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name=cal WARNING:root:fastparquet package not available. Parquet files will not be written. CameraMapper INFO: Loading exposure registry from /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered/registry.sqlite3 meas.base.forcedPhotCcd.PerTractCcdDataIdContainer INFO: Reading WCS for components of dataId={'visit': 1993939} to determine tracts meas.base.forcedPhotCcd.PerTractCcdDataIdContainer INFO: Number of visits for each tract: {0: 1} root INFO: Running: /global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/bin/visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name=cal visitAnalysis FATAL: Failed in task initialization: Version mismatch (geom: 13.0 vs 13.0-1-g7257b6a+14; numpy.linalg._umath_linalg: 0.1.4 vs b'0.1.4'; pex_exceptions: 13.0 with boost=1.60.lsst1+1 vs 13.0-3-g13ef843+12 with boost=1.60.lsst1+1; urllib.request: 2.7 vs 3.5; future: 0.15.2 vs 0.16.0; past: 0.15.2 vs 0.16.0; pex_config: 13.0 with boost=1.60.lsst1+1 vs 13.0-4-gbcd7061+12 with boost=1.60.lsst1+1; ip_isr: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-15-g0af5a6c+28 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; afw: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-110-ge674a0930+1 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; log: 13.0 with boost=1.60.lsst1+1 vs 13.0-7-g73c05b9+11 with boost=1.60.lsst1+1; python: 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] vs 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]; distutils: 2.7.12 vs 3.5.2; meas_algorithms: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-24-g22030a45+11 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; meas_astrom: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-16-g30f6da5+28 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; pyfits.extern.six.moves.xmlrpc_client: 1.0.1 vs 3.5; ip_diffim: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-28-gf4bc96c+13 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; daf_persistence: 13.0 with boost=1.60.lsst1+1 vs 13.0-41-g024c5b6 with boost=1.60.lsst1+1; pyfits.extern.six.moves.BaseHTTPServer: 0.3 vs 0.6; coadd_utils: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-4-g4231ded+46 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; obs_lsstSim: 2017.02.08.twinkles-9-g02e891a with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs LOCAL:/global/cscratch1/sd/heatherk/qa_dm_11452/code/obs_lsstSim@701b7b45633cbe3f1e0c8550582c2be521f7b876; obs_base: 13.0 with boost=1.60.lsst1+1 vs 13.0-55-gb064ced+5 with boost=1.60.lsst1+1; display_ds9: 13.0 with boost=1.60.lsst1+1 vs 13.0-2-g6174d4c+66 with boost=1.60.lsst1+1; pex_policy: 13.0 with boost=1.60.lsst1+1 vs 13.0-2-g4b114ac+12 with boost=1.60.lsst1+1; base: 13.0 vs 13.0-12-g6586b4b+4; daf_base: 13.0 with boost=1.60.lsst1+1 vs 13.0-5-g99bcfa4+9 with boost=1.60.lsst1+1; utils: 13.0 with boost=1.60.lsst1+1 vs 13.0-12-gac01bb5+7 with boost=1.60.lsst1+1; numpy.core.umath: 0.4.0 vs b'0.4.0'; meas_deblender: 13.0 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2 vs 13.0-13-g8b7e855+41 with boost=1.60.lsst1+1 eigen=3.2.5.lsst2-1-g8975fe8; galsim: 1.4.1 vs 1.4.3; pyfits.extern.six.moves.CGIHTTPServer: 0.4 vs 0.6); consider using --clobber-versions or --no-versions Thinking this may be due to my work installing obs_lsstSim at NERSC for DC1, I eliminated my $HOME/.eups directory and tried again and again :). That error persists. This is a fresh login, with only w_2017_41 in my env so I'm at a loss where 13.0 and py2.7 is coming from. python --version Python 3.5.2 :: Continuum Analytics, Inc. heatherk@cori03:/global/cscratch1/sd/heatherk/qa_dm_11452> which python /global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.2.12/bin/python I also tried the --clobber-version option, which results in a completely different error: heatherk@cori03:/global/cscratch1/sd/heatherk/qa_dm_11452> visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun $SCRATCH/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name=cal --clobber-versions WARNING:root:fastparquet package not available. Parquet files will not be written. CameraMapper INFO: Loading exposure registry from /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered/registry.sqlite3 meas.base.forcedPhotCcd.PerTractCcdDataIdContainer INFO: Reading WCS for components of dataId={'visit': 1993939} to determine tracts meas.base.forcedPhotCcd.PerTractCcdDataIdContainer INFO: Number of visits for each tract: {0: 1} root INFO: Running: /global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/bin/visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name=cal --clobber-versions visitAnalysis INFO: dataRefList size: 28 visitAnalysis FATAL: Failed on dataId=[DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,0', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,0', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,0', 'raft': '1,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,0', 'raft': '1,4'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,1', 'raft': '0,1'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,1', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,1', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,1', 'raft': '1,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,1', 'raft': '1,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,2', 'raft': '0,1'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,2', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,2', 'raft': '1,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '0,2', 'raft': '1,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,0', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,0', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,1', 'raft': '0,1'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,1', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,1', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,2', 'raft': '0,1'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,2', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '1,2', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,0', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,0', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,1', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,1', 'raft': '0,3'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,2', 'raft': '0,1'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,2', 'raft': '0,2'}, tag=set()), DataId(initialdata={'visit': 1993939, 'tract': 0, 'filter': 'r', 'sensor': '2,2', 'raft': '0,3'}, tag=set())]: 'DataId' object has no attribute 'has_key' Traceback (most recent call last): File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+35/python/lsst/pipe/base/cmdLineTask.py", line 407, in __call__ result = task.run(dataRef, **kwargs) File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/visitAnalysis.py", line 206, in run repoInfo = getRepoInfo(dataRefListTract[0], doApplyUberCal=self.config.doApplyUberCal) File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/utils.py", line 867, in getRepoInfo isCoadd = True if dataId.has_key("patch") else False AttributeError: 'DataId' object has no attribute 'has_key' Is the above a legitimate error @jchiang87 @laurenam ? I'm open to the idea that there something in my env which continues to point at 13.0 (that is a version we have installed at NERSC) but I can't explain the missing attribute "has_key". In the JIRA, @laurenam had first generated the cal_ref_cat directories using ingestReferenceCatalog.py - we have that directory at NERSC, but perhaps the format is slightly different for these updates?
@jamesp-epcc You could try to install your own copies of obs_lsstSim and pipe_analysis using those same directions in the DM JIRA linked in one of the previous comments, using the w_2017_41 we have available and see if you encounter a similar result.

[jamesp-epcc]

I just tried and I get the same errors as @heather999 . All I did beforehand was to source the setup_w_2017_41.sh script and run "setup lsst_distrib".

[heather999]

Out of curiosity @jamesp-epcc if you run visitAnalysis.py without the --clobber-version parameter, do you also receive an error about version mismatches?

[jamesp-epcc]

@heather999 yes. Without the parameter I get the version mismatch error (though the list of mismatches appears to be in a slightly different order from yours), with --clobber-versions I get the "has_key" error.

[heather999]

Thanks @jamesp-epcc, that helps. Maybe I should be looking in the desc account's environment. Ignoring that for a moment, given the error we see with the --clobber-version parameter, I'm inclined to try to reproduce the cal_ref_cat directories, but I cannot find the config file mentioned in the DM JIRA. ingestReferenceCatalog.py /scratch/lauren/rerun/DC1/ -C config/IngestIndexedReferenceTask.py

[laurenam]

This looks like a python3 incompatibility (apparently has_key() has been removed). Give me a minute and I'll push an updated branch.

[r-owen]

Good catch. has_key has been deprecated for some time; foo in myDict is the preferred alternative to myDict.has_key(foo) even in Python 2.

[laurenam]

Sorry it took so long...it took me a while to figure out the updates needed in the mapper file (among a few other py3-isms!). So, there are updated tickets/DM-11452 branches pushed to both obs_lsstSim and pipe_analysis that should now work...let me know if they don't!

I should also note that I typically run these scripts with --no-versions to avoid the issues you noted above. Sorry for leaving that out...

[heather999]

Thank you @laurenam. It's getting better! I reinstalled obs_lsstSim and pipe_analysis and tried a new run, a couple of plots are generated, and then we receive a new error: visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name=cal --no-versions WARNING:root:fastparquet package not available. Parquet files will not be written. CameraMapper INFO: Loading exposure registry from /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered/registry.sqlite3 meas.base.forcedPhotCcd.PerTractCcdDataIdContainer INFO: Reading WCS for components of dataId={'visit': 1993939} to determine tracts meas.base.forcedPhotCcd.PerTractCcdDataIdContainer INFO: Number of visits for each tract: {0: 1} root INFO: Running: /global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/bin/visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name=cal --no-versions visitAnalysis INFO: dataRefList size: 28 visitAnalysis INFO: dataId: DataId(initialdata={'sensor': '0,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()) visitAnalysis INFO: Exising data for tract 0: ccdListPerTract = ['1300', '0211', '0101', '0102', '0320', '0201', '0222', '0210', '0111', '0311', '1201', '0212', '0200', '0322', '1302', '1400', '0202', '0300', '1202', '0312', '0321', '0310', '0122', '0221', '1301', '0220', '0112', '0301'] visitAnalysis INFO: Using 2.5*log10(FLUXMAG0) = 31.8045 from FITS header for zeropoint visitAnalysis INFO: shortName = footNpixHist visitAnalysis INFO: shortName = footNpix_calib_psfUsed visitAnalysis INFO: No data for dataset: galaxy visitAnalysis FATAL: Failed on dataId=[DataId(initialdata={'sensor': '0,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '0,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '0,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '1,3'}, tag=set()), DataId(initialdata={'sensor': '0,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '1,4'}, tag=set()), DataId(initialdata={'sensor': '0,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,1'}, tag=set()), DataId(initialdata={'sensor': '0,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '0,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '0,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '1,2'}, tag=set()), DataId(initialdata={'sensor': '0,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '1,3'}, tag=set()), DataId(initialdata={'sensor': '0,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,1'}, tag=set()), DataId(initialdata={'sensor': '0,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '0,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '1,2'}, tag=set()), DataId(initialdata={'sensor': '0,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '1,3'}, tag=set()), DataId(initialdata={'sensor': '1,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '1,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '1,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,1'}, tag=set()), DataId(initialdata={'sensor': '1,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '1,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '1,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,1'}, tag=set()), DataId(initialdata={'sensor': '1,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '1,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '2,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '2,0', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '2,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '2,1', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set()), DataId(initialdata={'sensor': '2,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,1'}, tag=set()), DataId(initialdata={'sensor': '2,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,2'}, tag=set()), DataId(initialdata={'sensor': '2,2', 'filter': 'r', 'visit': 1993939, 'tract': 0, 'raft': '0,3'}, tag=set())]: Unknown property facecolor Traceback (most recent call last): File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-14-g8b3bf66+35/python/lsst/pipe/base/cmdLineTask.py", line 407, in __call__ result = task.run(dataRef, **kwargs) File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/visitAnalysis.py", line 244, in run highlightList=[("parent", 0, "yellow"), ]) File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 539, in plotFootprint highlightList=highlightList) File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/visitAnalysis.py", line 40, in plotAll postFix=postFix, plotRunStats=plotRunStats, highlightList=highlightList) File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 730, in plotAll zpLabel=zpLabel, forcedStr=forcedStr, dataName="star") File "/global/cscratch1/sd/heatherk/qa_dm_11452/code/pipe_analysis/python/lsst/pipe/analysis/analysis.py", line 471, in plotSkyPosition fig, axes = plt.subplots(1, 1, subplot_kw=dict(facecolor="0.35")) File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.2.12/lib/python3.5/site-packages/matplotlib/pyplot.py", line 1186, in subplots ax0 = fig.add_subplot(gs[0, 0], **subplot_kw) File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.2.12/lib/python3.5/site-packages/matplotlib/figure.py", line 1005, in add_subplot a = subplot_class_factory(projection_class)(self, *args, **kwargs) File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.2.12/lib/python3.5/site-packages/matplotlib/axes/_subplots.py", line 73, in __init__ self._axes_class.__init__(self, fig, self.figbox, **kwargs) File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.2.12/lib/python3.5/site-packages/matplotlib/axes/_base.py", line 537, in __init__ self.update(kwargs) File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.2.12/lib/python3.5/site-packages/matplotlib/artist.py", line 856, in update raise AttributeError('Unknown property %s' % k) AttributeError: Unknown property facecolor

[laurenam]

Oh...this looks like a matplotlib version incompatibility. With the shared stack I'm using, the version is 2.0.2, and it looks like facecolor was added in 2.0.0. I'm guessing you've got an older version?

[heather999]

Right - for simplicity, I've stuck with the python that the stack installs, which leaves us at matplotlib 1.5.1. I don't know why they have such an old version. For this exercise we could upgrade, though I was hoping to set this up as a fairly vanilla stack in case we use it for other purposes. Let me do a little inquiry into why we're at matplotlib 1.5.1 when installing w.2107.41.

[heather999]

I'm re-installing w.2017.41 to get an updated python environment. Back in a couple hours :) https://community.lsst.org/t/stack-installed-miniconda/2362/2

[heather999]

The build is still chugging - due to where I'm building at NERSC, it has to build on the login nodes, which likely slows this process down a bit. Let's assume that all completes successfully and we can run the scripts. After that, the first step, I imagine, is to re-do the same test @laurenam completed to show we get the same plots and can run the comparison scripts. After that, we would want to continue testing on additional DC1 data - any suggestions @jchiang87?

[jchiang87]

@cwwalter may have some suggestions. One thing to try would be to rerun some parts of the L2 analysis using v14 on the DC1 dithered data and compare the output vs the original DC1 analysis. Single visits would be relatively easy...just re-running processEimage.py I imagine for some selection of visits. Comparing coadd results would mean running the L2 pipeline at least as far as we did for the DC1 analysis. If the DC1 phosim dithered data are still available, we could run these scripts on those data. Ultimately, we want the ability to run these scripts on the L2 output running on the protoDC2 dataset.

[heather999]

Things are looking good. The w.2017.41 build is complete, and I installed obs_lsstSim and pipe_analysis and am able to run visitAnalysis and compareVisitAnalysis. I notice one warning: WARNING:root:fastparquet package not available. Parquet files will not be written. Do we want to install fastparquet? Is it this one @laurenam ? https://pypi.python.org/pypi/fastparquet

I will install these versions of obs_lsstSim and pipe_analysis as part of the desc installation in the common software area, so we can all use the same version. On the side, I'll get a KNL installation going also using gcc for now.

I'm going to claim a lot of ignorance about the L2 processing and DC1 data generally @jchiang87 but your high-level summary has gotten me started on looking at processEimage in the obs_lsstSim repo and the SSim_DC1 repo. I might poke @tony-johnson for additional details about running the L2 pipeline as we did for DC1. For single visits, are some better than others - is there a list @cwwalter or are they all deemed suitable? I suspect I should ask @TomGlanzman about the DC1 phosim dithered data.

[jchiang87]

Looking back at https://jira.lsstcorp.org/browse/DM-11452, I think once you've demonstrated that you can reproduce the runs that Lauren did, this particular issue can be closed (unless Chris wants any other additional QA runs done).

I think learning how to run the current L2 pipeline would be a great thing to work on next, since we'll need to do that for the protoDC2 analysis. I'll make a new issue and we can carry on that discussion there.

[cwwalter]

Yes, to suggestions above. First rerunning @laurenam 's tests and looking at them, understanding them and making sure they make sense.

Then you will find out some of what Lauren wanted to do she couldn't because our stack is too old. So it would be good to re run with v14+.

Then there are new modules which we didn't include (like the one that returns deblendeness) we should have. It can be a warm up to possibly redoing the processing on at least imSim dithered with v14+.

Thanks!

[laurenam]

Glad to hear things are now working! @heather999, the Parquet tables are for use with an interactive and larger scale QA interface (Bokeh-based) that @timothydmorton is working on (see JIRA tickets DM-10619, DM11682, DM-12030, etc. for details). If you are interested, I'd ping him...I know he is looking for some guinea pigs to give the infrastructure a test drive, so you could probably twist his arm for a demo. I wouldn't bother writing out the Parquet tables before talking with him (and not before DM-12038 gets merged to pipe_analysis), but this is going to be a very powerful infrastructure beyond what the static pipe_analysis scripts provide, so you certainly can (and should!) check it out :)

Speaking of merging, I will merge DM-11452 as soon as you are all satisfied the plotting scripts are working on the DC1 data. I will put it in review to Heather, but feel free to reassign if desired. BTW, while the reviewer is welcome to make specific comments on the code, this is not strictly required here as the repo still lives in lsst-dm (if/when it gets moved to the main lsst repo, it will likely get a proper coding standards & practices review). The requirement here is simply a confirmation that you are able to run the scripts as advertised.

[cwwalter]

BTW I don't know much about Parquet but I know it is now supported in the latest version of Pandas too. The pandas authors are fans of it.

[timothydmorton]

If you do want to write the parquet tables (best to do after DM-12038 is merged), then pip install fastparquet should enable it. And yes, do ping me if you want to test out the interactive QA stuff.

[jamesp-epcc]

The scripts are working for me as well now, since Heather updated the installation. I have compared the plots generated to the ones Lauren attached on the Jira ticket. They look similar but not identical. I don't know enough about the science to know whether the differences are acceptable or not, but if anyone wants to take a look, the plots I generated are in /global/cscratch1/sd/jamesp/dc1rerun/plots on Cori.

[laurenam]

If the data going into the scripts are the same, the plots produced should also be the same. If you put your figures somewhere I can see them, I'll have a look...

[heather999]

Updated Comment: Here is an example of a plot produced by visitAnalysis that reports different values from what we see on the JIRA. My immediate question, is whether I used the same parameters, I didn't place any specific parameters for the visit, other than the visit id. https://confluence.slac.stanford.edu/display/LSSTDESC/QA+Testing+plots

Is it sensible to try a run of compareCoaddAnalysis and compareVisitAnalysis using your data posted to the JIRA @laurenam run against the visitAnalysis and coaddAnalysis data we produce using the same visit as input? I've just tried to do that and have a pile of plots. Just starting to go through them. I did things like compareVisitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --rerun2 /global/cscratch1/sd/heatherk/qa_dm_11452/jira --id visit=1993939 --tract=0 --config doApplyUberCal1=False doApplyUberCal2=False --config refObjLoader.ref_dataset_name="cal_ref_cat" --no-versions and compareCoaddAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --rerun2 /global/cscratch1/sd/heatherk/qa_dm_11452/jira --id tract=0 patch=18,13 filter=r --config refObjLoader.ref_dataset_name="cal_ref_cat" --no-versions where the jira directory contains the tarballs posted to the JIRA. I posted the resulting plots and a tarball (AllPlots.tar.gz) to a Confluence page - due to the large number of files, the page loads slowly for me.. I'll see if I can provide an easier way to view them. https://confluence.slac.stanford.edu/pages/viewpageattachments.action?pageId=228301438

[heather999]

Would our use of py3 rather than py2 perhaps introduce some differences? I'm assuming the tests on lsat-dev used python 2?

[laurenam]

We have certainly seen a few py3 vs. py2 differences in some processing. I'm running the scripts using py3 now to see if I see any differences.

BTW, I don't have access to SLAC's confluence, so I can't see your plots...

[heather999]

I was wondering about that @laurenam Where would it be convenient to drop a tarball? I could come back to the DM JIRA, perhaps.. I'm not sure if I have access to lsst-dev, let me see..

[laurenam]

I was able to assign you as reviewer...I would hope that implies you have access!!

[laurenam]

I'm not seeing any differences so far. Can you tell me the name of one of the plots where you see differences (or is it all of them?). Perhaps just post a few examples (i.e. not the whole tarball) to JIRA (if you get in, that is)?

[heather999]

sorry for the delay, yes, I'll do that asap.

[cwwalter]

Could we get Heather lsst-dev access? Looking at https://jira.lsstcorp.org/browse/DM-11452 that could help this work I think.

[heather999]

Actually I received an lsst-dev account a few weeks ago.. I'm working on trying it out now. I just don't know anything about where things are, but I can inquire. In the meantime, I'll respond on the DM JIRA with some ideas. I could do it here too - I'm not sure where is best to get the right eyes looking at the commentary and to avoid too much confusion about where we've put various comments. #TooManyCommunicationTools :)

[heather999]

@jamesp-epcc, you can view the DM JIRA too? Just want to make sure!

[jamesp-epcc]

@heather999 yes, I can view it. I don't think I have an account as I see a "Log In" button but it seems to be possible to view without one.

[cwwalter]

I'm not sure where is best to get the right eyes looking at the commentary and to avoid too much confusion about where we've put various comments. #TooManyCommunicationTools :)

I think doing things here (GH) generically is probably better. For project related communication Jira is best. Most people can't comment there because they don't have project credentials and it isn't that well known to DESC people.

If there is a compelling reason to do it on Jira (related to project decision making or some Jira feature related to attachment etc that are needed) of course feel free to use it. But let's default to here for the DESC work.

[heather999]

Just to bring people up to date here.. after copying Jim's data from NCSA back to NERSC (where the data originally came from) we now have agreement between running the scripts at NERSC and at NCSA. I've indicated that the review is complete on the JIRA. I'll go back through the comments above to make sure we capture any bits necessary in other issues.

[cwwalter]

In the spirit of trying to consolidate DESC related info, I wanted to copy some information to here from the associated JIRA ticket https://jira.lsstcorp.org/browse/DM-11452. This is about what information we would get added to the output by running at least V14 of the stack. Several of the things @laurenam checks were not in V13. This is also relevant to #47.

Chris Walter, most of the accommodations & plot skipping were based on flags not in the old schemas. A new reprocessing should get them by default.
Have a look here for the plugins specifically added in obs_subaru:
https://github.com/lsst/obs_subaru/blob/master/config/processCcd.py#L70-L82
(and all the associated loaded config files)
https://github.com/lsst/obs_subaru/blob/master/config/processCcd.py#L91-L92
and for coadd measurement, have a look at:
https://github.com/lsst/obs_subaru/blob/master/config/measureCoaddSources.py