search

transientskp / tkp

A transients-discovery pipeline for astronomical image-based surveys
http://docs.transientskp.org/
BSD 2-Clause "Simplified" License
19 stars 15 forks source link

TraP sourcefinder error on updated AARTFAAC images #505

Closed ycendes closed 8 years ago

ycendes commented 8 years ago

I got the following error over the weekend running the new AARTFAAC images through TraP. As this is PhD thesis dependent, I'm hoping it can be fixed ASAP-

No handlers could be found for logger "tkp.steps.quality"
18:36:29:tkp.main:INFO processing 1 images in timestep 2013-11-20 11:11:01.006080 (1/82199)
Traceback (most recent call last):
  File "/soft/trap/r3.1.0/bin/http://trap-manage.py", line 5, in <module>
    pkg_resources.run_script('tkp==3.1.0', 'http://trap-manage.py')
  File "/usr/lib/python2.7/dist-packages/http://pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/http://pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/EGG-INFO/scripts/http://trap-manage.py", line 10, in <module>
    tkp.management.main()
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/http://management.py", line 329, in main
    args.func(args)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/http://management.py", line 227, in run_job
    run(http://args.name, monitor_coords)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/http://main.py", line 142, in run
    extraction_results = runner.map("extract_sources", urls, arguments)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/distribute/http://__init__.py", line 42, in map
    return self.module.map(func, iterable, args)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/distribute/multiproc/http://__init__.py", line 23, in map
    return pool.map(func, zipped)
  File "/usr/lib/python2.7/multiprocessing/http://pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/http://pool.py", line 558, in get
    raise self._value
AssertionError

I reran the TraP on the same data set but in serial mode, and got this:

16:09:28:tkp.main:INFO processing 1 images in timestep 2013-11-20 11:11:01.006080 (1/82199)
Traceback (most recent call last):
  File "/soft/trap/r3.1.0/bin/http://trap-manage.py", line 5, in <module>
    pkg_resources.run_script('tkp==3.1.0', 'http://trap-manage.py')
  File "/usr/lib/python2.7/dist-packages/http://pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/http://pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/EGG-INFO/scripts/http://trap-manage.py", line 10, in <module>
    tkp.management.main()
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/http://management.py", line 329, in main
    args.func(args)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/http://management.py", line 227, in run_job
    run(http://args.name, monitor_coords)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/http://main.py", line 142, in run
    extraction_results = runner.map("extract_sources", urls, arguments)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/distribute/http://__init__.py", line 42, in map
    return self.module.map(func, iterable, args)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/distribute/serial/http://__init__.py", line 3, in map
    x = [func(i, *arguments) for i in iterable]
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/distribute/serial/http://tasks.py", line 22, in extract_sources
    return tkp.steps.source_extraction.extract_sources(url, extraction_params)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/steps/http://source_extraction.py", line 49, in extract_sources
    force_beam=extraction_params['force_beam']
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/http://image.py", line 404, in extract
    det * self.rmsmap, anl * self.rmsmap, deblend_nthresh, force_beam,
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/utility/http://memoize.py", line 29, in __call__
    self.memo[instance] = self.funct(instance)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/http://image.py", line 114, in _get_rm
    return self._interpolate(self.grids['rms'], roundup=True)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/utility/http://memoize.py", line 29, in __call__
    self.memo[instance] = self.funct(instance)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/http://image.py", line 92, in _grids
    return self.__grids()
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/http://image.py", line 222, in __grids
    assert(len(useful_chunk) == 1)
AssertionError

The log files for this last run are at /scratch/ycendes/new-aartfaac/new-aartfaac/cendes-new-aartfaac/logs/2016-04-25T09:54:51 and are attached. error-logs.zip

ycendes commented 8 years ago

Error from pyse.py, and the first image of the batch is attached.

ycendes@struis:/scratch/ycendes/new-aartfaac/new-aartfaac/cendes-new-aartfaac/logs/2016-04-25T09:54:51$ pyse.py --detection=8 --analysis=3 /scratch/fhuizing/24h/S294_C63_M9_T20131120-101100/S294_R0-62_T20-11-2013_10-11-01.image
Processing /scratch/fhuizing/24h/S294_C63_M9_T20131120-101100/S294_R0-62_T20-11-2013_10-11-01.image (file 1 of 1).
Thresholding with det = 8.000000 sigma, analysis = 3.000000 sigma
Traceback (most recent call last):
  File "/soft/trap/r3.1.0/bin/pyse.py", line 5, in <module>
    pkg_resources.run_script('tkp==3.1.0', 'pyse.py')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/EGG-INFO/scripts/pyse.py", line 358, in <module>
    print run_sourcefinder(files, options),
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/EGG-INFO/scripts/pyse.py", line 319, in run_sourcefinder
    force_beam=options.force_beam
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/image.py", line 404, in extract
    det * self.rmsmap, anl * self.rmsmap, deblend_nthresh, force_beam,
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/utility/memoize.py", line 29, in __call__
    self.memo[instance] = self.funct(instance)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/image.py", line 114, in _get_rm
    return self._interpolate(self.grids['rms'], roundup=True)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/utility/memoize.py", line 29, in __call__
    self.memo[instance] = self.funct(instance)
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/image.py", line 92, in _grids
    return self.__grids()
  File "/soft/trap/r3.1.0/lib/python2.7/site-packages/tkp-3.1.0-py2.7.egg/tkp/sourcefinder/image.py", line 222, in __grids
    assert(len(useful_chunk) == 1)
AssertionError

S294_R0-62_T20-11-2013_10-11-01.image.zip

gijzelaerr commented 8 years ago

this has been fixed and is released.