search

mwvgroup / wiyn_lsst

Processing of SweetSpot WIYN+WHIRC images using LSST DM Stack.
Other
0 stars 0 forks source link

The filename mapper is not figuring out what to call the file in subtraction routine #9

Closed kponder closed 6 years ago

kponder commented 7 years ago

The subtraction routine is not producing the final diffexp.fits file. It is creating a tmp/test_dr2/diff directory and adding subdirectories for supernovae that we calibrated, but there are never the final images. Here is a snippet of error from SN2011iu. The stacks are all good for this one and it is calibrated.

It makes the temporary fits files that it deposits in /tmp/test_dr2/, but never deletes them or creates the /tmp/test_dr2/diff/$supernova$/diffexp.fits files from them.

From MWV: " There's something wrong with filename mapper. It's not figuring out what to call the file. Will have to think about this."

Error from Cori:

[cori03 wiyn_lsst] python sub_wiyn_dr2.py $SCRATCH/tmp/test_dr2              
Processing SN2011iu                                                          
Running ImageDifferenceTask.parseAndrun with args:                           
['/global/cscratch1/sd/kap146/tmp/test_dr2', '--id', 'fileroot=SN2011iu_A_H_20120108', '--templateId', 'fileroot=SN2011iu_A_H_20120925', '--output', '/global/cscratch1/sd/kap146/tmp/test_dr2', '--configfile', 'diffimconfig.py', '--clobber-config', '--clobber-versions']                                                 
root INFO: Running: lsst.pipe.base.cmdLineTask.ImageDifferenceTask.parseAndRun('/global/cscratch1/sd/kap146/tmp/test_dr2', '--id', 'fileroot=SN2011iu_A_H_20120108', '--templateId', 'fileroot=SN2011iu_A_H_20120925', '--output', '/global/cscratch1/sd/kap146/tmp/test_dr2', '--configfile', 'diffimconfig.py', '--clobber-config', '--clobber-versions')                                                                             
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.                       
Use '--' to separate paths from revisions, like this:                                                     
'git <command> [<revision>...] -- [<file>...]'                                                            
fatal: Not a git repository: '/global/homes/w/wmwv/local/lsst/obs_file/.git'                              
imageDifference INFO: Processing DataId(initialdata={'fileroot': 'SN2011iu_A_H_20120108'}, tag=set([]))   
imageDifference.getTemplate INFO: Fetching calexp (DataId(initialdata={'fileroot': 'SN2011iu_A_H_20120925'}, tag=set([]))) as template.                                                                            
imageDifference INFO: Source selection via src product                                                    
/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/meas_algorithms/12.1-17-g13cfda1+4/python/lsst/meas/algorithms/objectSizeStarSelector.py:399: RuntimeWarning: invalid value encountered in less                 
  bad = numpy.logical_or(bad, width < self.config.widthMin)                                               
/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/meas_algorithms/12.1-17-g13cfda1+4/python/lsst/meas/algorithms/objectSizeStarSelector.py:400: RuntimeWarning: invalid value encountered in greater              
  bad = numpy.logical_or(bad, width > self.config.widthMax)                                               
imageDifference INFO: Selected 32 / 534 sources for Psf matching (8 for control sample)                   
imageDifference INFO: Subtracting images                                                                  
imageDifference.subtract INFO: Template Wcs : 6.245210,0.816067 -> 6.243156,0.814774                      
imageDifference.subtract INFO: Science Wcs : 6.245101,0.816153 -> 6.243204,0.814880                       
imageDifference.subtract INFO: Astrometrically registering template to science image                      
imageDifference.subtract INFO: templateFwhmPix: 4.73901486328                                             
imageDifference.subtract INFO: scienceFwhmPix: 11.8430608219                                              
imageDifference.subtract INFO: Growing 32 kernel candidate stars by 35 pixels                             
imageDifference.subtract INFO: Selected 32 / 32 sources for KernelCandidacy                               
imageDifference.subtract INFO: Matching Psf FWHM 4.74 -> 11.84 pix                                        
imageDifference.subtract INFO: Final spatial kernel sum 1.120                                             
imageDifference.subtract INFO: Spatial model condition number 3.566e+07                                   
imageDifference.subtract INFO: Doing stats of kernel candidates used in the spatial fit.                  
imageDifference.subtract INFO: 32 candidates total, 18 rejected, 14 used                                  
imageDifference.subtract INFO: Spatial kernel model well constrained; 14 candidates, 3 terms, 27 bases    
imageDifference.subtract INFO: Spatial background model appears well constrained; 14 candidates, 1 terms  
imageDifference FATAL: Failed on dataId=DataId(initialdata={'fileroot': 'SN2011iu_A_H_20120108'}, tag=set([])): [Errno 20] Not a directory                                                                         
Traceback (most recent call last):                                                                        
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/pipe_base/12.1-5-g06c326c+4/python/lsst/pipe/base/cmdLineTask.py", line 347, in __call__                                                                
    result = task.run(dataRef, **kwargs)                                                                  
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/pipe_base/12.1-5-g06c326c+4/python/lsst/pipe/base/timer.py", line 121, in wrapper                                                                      
    res = func(self, *args, **keyArgs)                                                                    
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/pipe_tasks/12.1-20-g324f6d3+4/python/lsst/pipe/tasks/imageDifference.py", line 521, in run                                                              
    sensorRef.put(subtractRes.matchedExposure, self.config.coaddName + "Diff_matchedExp")                 
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/butlerSubset.py", line 216, in put                                                       
    self.butlerSubset.butler.put(obj, datasetType, self.dataId, doBackup=doBackup, **rest)                
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/butler.py", line 784, in put                                                             
    location.getRepository().write(location, obj)                                                         
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/repository.py", line 144, in write                                                       
    return self._storage.write(butlerLocation, obj)                                                       
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/posixStorage.py", line 229, in write                                                    
    self.persistence.persist(obj, storageList, additionalData)                                            
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/miniconda2/3.19.0.lsst4/lib/python2.7/contextlib.py", line 24, in __exit__                                                                              
    self.gen.next()                                                                                       
  File "/global/common/cori/contrib/lsst/lsstDM/dc1/Linux64/daf_persistence/12.1-17-g9654cba+4/python/lsst/daf/persistence/safeFileIo.py", line 126, in SafeFilename                                               
    os.rename(tempName, name)                                                                             
OSError: [Errno 20] Not a directory
wmwv commented 6 years ago

No longer a problem in the updated obs_wiyn