search

mwvgroup / wiyn_lsst

Processing of SweetSpot WIYN+WHIRC images using LSST DM Stack.
Other
0 stars 0 forks source link

Create per field/filter coadds for all DR1 fields #13

Closed wmwv closed 6 years ago

wmwv commented 6 years ago
  1. [x] Create coadd for one field
  2. [x] Do photometry for one field
  3. [x] Create coadds for all fields. -> #17 Blocked on #16.
  4. [x] Created photometry for all fields. -> #17 Blocked on #16
wmwv commented 6 years ago

Done for LSQ13cwp. Next PTF 11mty.

wmwv commented 6 years ago

PTF 11mty processes. All 6 J, H images generate reasonable-seeming zeropoints.

wmwv commented 6 years ago

makeDiscreteSkyMap.py looks like what I want to use. It generates a map for a specific location (appropriate for a field-targeted survey), rather than a sky tiling (appropriate for a large-area survey).

But for each field, I need to append the creation of the field using --config doAppend=True

wmwv commented 6 years ago

Could also consider using makeDiscreteSkyMapRunner which is designed to accept an entire set of dataRefs at once.

wmwv commented 6 years ago

Successfully generated H-band, J-band coadds for PTF11mty.

wmwv commented 6 years ago

Had to specify which field to sort on in /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/skymap/14.0-4-g3609236+4/python/lsst/skymap/baseSkyMap.py, line 129:

        distTractInfoList.sort(key=lambda x: x[0])
wmwv commented 6 years ago

Now failing trying to look up some information about the original Visit?

measureCoaddSources.propagateFlags INFO: Propagating flags dict_keys(['calib_psfCandidate', 'calib_psfUsed']) from inputs
measureCoaddSources FATAL: Failed on dataId=DataId(initialdata={'tract': 1, 'patch': '0,0', 'field': 'PTF11mty', 'filter': 'J'}, tag=set()): OperationalError: no such table: raw_visit
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
    result = task.run(dataRef, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/multiBand.py", line 1098, in run
    exposure.getWcs())
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/propagateVisitFlags.py", line 184, in run
    ccdSources = butler.get("src", visit=int(v), ccd=int(c), immediate=True)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1408, in get
    location = self._locate(datasetType, dataId, write=False)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1329, in _locate
    location = repoData.repo.map(datasetType, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/repository.py", line 246, in map
    loc = self._mapper.map(*args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/mapper.py", line 166, in map
    return func(self.validate(dataId), write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/cameraMapper.py", line 383, in mapClosure
    return mapping.map(mapper, dataId, write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 142, in map
    actualId = self.need(iter(self.keyDict.keys()), dataId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 278, in need
    lookups = self.lookup(newProps, newId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 242, in lookup
    result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/registries.py", line 369, in lookup
    cursor.execute(cmd, valueList)
sqlite3.OperationalError: no such table: raw_visit
The program exited via sys.exit(). Exit status: 2
> /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/bin/measureCoaddSources.py(2)<module>()
-> from lsst.pipe.tasks.multiBand import MeasureMergedCoaddSourcesTask
wmwv commented 6 years ago

The issue above is that PropagateVisitFlagsTask assumes that there is a ccd entry in the raw (or in this case stack) table. This is non-crazy, but not documented or otherwise enforced in the ingest step.

wmwv commented 6 years ago

  1. I changed the src and transformed_src definitions so that tables: stack instead of tables: raw.

  2. With some debugging print statements added, the relevant output is now:

REPODATA, DATASETTYPE, DATAID
RepoData(id=139934254811904,repoArgs=RepositoryArgs(root='/home/lsst/SweetSpot/DR1_data/tmp/test_dr1', cfgRoot=None, mapper=<class 'lsst.obs.wiyn.whircMapper.WhircMapper'>, mapperArgs=None, tags=set(), mode='rw', policy=None)cfg=RepositoryCfg(root='/home/lsst/SweetSpot/DR1_data/tmp/test_dr1', mapper=<class 'lsst.obs.wiyn.whircMapper.WhircMapper'>, mapperArgs={}, parents=[], policy=None),cfgOrigin=existing,cfgRoot=/home/lsst/SweetSpot/DR1_data/tmp/test_dr1,repo=<lsst.daf.persistence.repository.Repository object at 0x7f44fb8cc588>,parentRepoDatas=[],isV1Repository=True,role=output,parentRegistry=None) src DataId(initialdata={'visit': 0, 'ccd': -1}, tag=set())
CMD, VALUELIST
('SELECT DISTINCT field, seq, filter, night FROM stack_visit WHERE visit = ? AND ccd = ?', [0, -1])
measureCoaddSources FATAL: Failed on dataId=DataId(initialdata={'tract': 1, 'patch': '0,0', 'field': 'PTF11mty', 'filter': 'H'}, tag=set()): OperationalError: no such column: ccd
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
    result = task.run(dataRef, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/multiBand.py", line 1098, in run
    exposure.getWcs())
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/propagateVisitFlags.py", line 182, in run
    for ccdRecord in ccdInputs:
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1410, in get
    location = self._locate(datasetType, dataId, write=False)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/butler.py", line 1331, in _locate
    location = repoData.repo.map(datasetType, dataId, write=write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/repository.py", line 246, in map
    loc = self._mapper.map(*args, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/mapper.py", line 166, in map
    return func(self.validate(dataId), write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/cameraMapper.py", line 383, in mapClosure
    return mapping.map(mapper, dataId, write)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 142, in map
    actualId = self.need(iter(self.keyDict.keys()), dataId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 278, in need
    lookups = self.lookup(newProps, newId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_base/14.0-24-g714dd46+3/python/lsst/obs/base/mapping.py", line 242, in lookup
    result = self.registry.lookup(properties, self.tables, lookupDataId, template=self.template)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+11/python/lsst/daf/persistence/registries.py", line 371, in lookup
    cursor.execute(cmd, valueList)
sqlite3.OperationalError: no such column: ccd
wmwv commented 6 years ago

I will try skipping PropagateVisitFlags for now with --doPropagateFlags=False

wmwv commented 6 years ago

Following instructions at https://pipelines.lsst.io/getting-started/photometry.html

For PTF11mty coadds have now successfully:

  1. Detected
  2. Merged Detections
  3. Measured
  4. Merged Measurements
  5. Forced photometry at merged object locations.
wmwv commented 6 years ago

The Forced photometry spews errors for all of the tracts that don't exist, but still appears to generate output for the one tract+patch that does exist.

wmwv commented 6 years ago

Coadds look good. Example for LSQ13cwp lsq13cwp_j

wmwv commented 6 years ago

mag err vs. mag looks reasonable. lsq13cwp_j_mag_err_mag

wmwv commented 6 years ago

Incorporated the deepCoadd_ref table, including star/galaxy separation information, but it's not very good. Obvious galaxies are marked as stars and vice versa.

wmwv commented 6 years ago

When running all of the fields, I was getting failures where processCcd.py was exiting. Because it was running 8 jobs in parallel it was a bit tough to diagnose exactly what was happening, but I think it's the K-band images that were failing. These are registered as filter KS. I'm re-running the full list without the K-band

wmwv commented 6 years ago

It looks like jobs are hanging on something. I see 9 python processes for each invocation of run_processCcd.sh. I think these aren't getting output to the log. Will debug in -j 1 tomorrow.

wmwv commented 6 years ago

Tried a sample image that previously failed: P161-D_A_KS_20121122

WIYN=${DR1BASE}/tmp
REPO=${WIYN}/test_dr1

processCcd.py "${REPO}" --rerun processCcdKS \
    --id field=P161-D seq=A filter=KS night=20121122 expnum=528 \
    --configfile config/processCcd.py

This failed as it had yesterday with:

processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 173/943 sources
processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 30/30 references
processCcd.calibrate.photoCal.match INFO: Matched 0 from 173/943 input and 30/30 reference sources
processCcd.calibrate.photoCal.reserve INFO: Reserved 0/0 sources
processCcd FATAL: Failed on dataId=DataId(initialdata={'field': 'P161-D', 'seq': 'A', 'filter': 'KS', 'night': 20121122, 'expnum': '528'}, tag=set()): RuntimeError: No matches to use for photocal
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
    result = task.run(dataRef, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/processCcd.py", line 199, in run
    icSourceCat=charRes.sourceCat,
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/calibrate.py", line 433, in run
    icSourceCat=icSourceCat,
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/calibrate.py", line 535, in calibrate
    photoRes = self.photoCal.run(exposure, sourceCat=sourceCat, expId=exposureIdInfo.expId)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+48/python/lsst/pipe/base/timer.py", line 150, in wrapper
    res = func(self, *args, **keyArgs)
  File "/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-52-g00a5fa4c/python/lsst/pipe/tasks/photoCal.py", line 464, in run
    raise RuntimeError("No matches to use for photocal")
RuntimeError: No matches to use for photocal

So this means the photometry calibration code is not finding the matches between the sources and the reference catalog.

wmwv commented 6 years ago

Looking more closely at the stack I saw that the WCS solution was slightly off by ~0.5 arcsec.

I ran it with astrometric solving and it succeeded:

processCcd.py "${REPO}" --rerun processCcdKS \
    --id field=P161-D seq=A filter=KS night=20121122 expnum=528 \
    --configfile config/processCcd.py \
    --config calibrate.doAstrometry=True
processCcd.calibrate.astromRefObjLoader INFO: Loading reference objects using center IcrsCoord(105.2138898, 48.4906848) and radius 0.06316164018914798 deg
processCcd.calibrate.astromRefObjLoader INFO: Loaded 33 reference objects
processCcd.calibrate.astrometry.matcher INFO: Purged 931 sources, leaving 12 good sources
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher WARN: Failed after 6 patterns.
processCcd.calibrate.astrometry.matcher INFO: Matched 12 sources
processCcd.calibrate.astrometry.matcher INFO: Purged 931 sources, leaving 12 good sources
processCcd.calibrate.astrometry.matcher INFO: Matched 12 sources
processCcd.calibrate.astrometry.matcher INFO: Purged 931 sources, leaving 12 good sources
processCcd.calibrate.astrometry.matcher INFO: Matched 12 sources
processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 12 matches with scatter = 0.004 +- 0.003 arcsec
processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 173/943 sources
processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 34/34 references
processCcd.calibrate.photoCal.match INFO: Matched 12 from 173/943 input and 34/34 reference sources
processCcd.calibrate.photoCal.reserve INFO: Reserved 0/12 sources
processCcd.calibrate.photoCal INFO: Not applying color terms because config.applyColorTerms is None and data is not available and photoRefCat is provided
processCcd.calibrate.photoCal INFO: Magnitude zero point: 26.380040 +/- 0.000343 from 12 stars
processCcd.calibrate INFO: Photometric zero-point: 26.380040
wmwv commented 6 years ago

I had previously disabled the astrometric fitting because it was failing to find matches for images that should have been trivially simple with WCS solutions that were already good to ~0.1 arcsec.

wmwv commented 6 years ago

Hmmm... indirectly related to the low success rate of processing only ~150 out of ~700 stacks last night (with no astrom fitting), I realize that I'm running this in a Docker VM with only 2GB. Running 8 simultaneous processing is probably too much.

wmwv commented 6 years ago

Fitting astrometry failed for P525-E_B_H_20131019

wmwv commented 6 years ago

I'm going to merge the present state to master. Then block on further progress until all stacks can be processed with processCcd.py #16 .