[AARTFAAC] TraP sourcefinder assertion fail on aartfaac-imaging-pipeline generated images

hsuyeep commented 9 years ago

Running on a sequence of 60 images from peeyush@struis:~/pipeline/testaartfaac/2013_pipeline_images/
The first 40 run through OK, but with warnings like: INFO:tkp.main:performing DB operations for image 40 INFO:tkp.main:performing source association INFO:tkp.steps.forced_fitting:Found 245 null detections INFO:tkp.steps.forced_fitting:Forced fitting in image: /home/peeyush/pipeline/testaartfaac/2013_pipeline_images/F5.71304e+07_S1-62_T20-11-2013_09-23-39.image WARNING:root:position errors extend outside image WARNING:root:position errors extend outside image WARNING:root:position errors extend outside image WARNING:root:position errors extend outside image WARNING:root:position errors extend outside image WARNING:root:position errors extend outside image WARNING:root:position errors extend outside image WARNING: <\snip>
Image 41 fails with an assertion fail: INFO:tkp.main:processing 1 images in timestep 2013-11-20 10:23:40.770240 (41/60) INFO:tkp.main:performing source extraction INFO:tkp.distribute.serial.tasks:running extracted sources task INFO:tkp.steps.source_extraction:Extracting image: /home/peeyush/pipeline/testaartfaac/2013_pipeline_images/F5.71304e+07_S1-62_T20-11-2013_09-23-40.image Traceback (most recent call last): File "/soft/trap/aartfaac/bin/trap-manage.py", line 10, in tkp.management.main() File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/management.py", line 335, in main args.func(args) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/management.py", line 229, in run_job run(args.name, monitor_coords) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/main.py", line 144, in run extraction_results = runner.map("extract_sources", urls, arguments) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/distribute/init.py", line 42, in map return self.module.map(func, iterable, args) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/distribute/serial/init.py", line 3, in map x = [func(i, arguments) for i in iterable] File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/distribute/serial/tasks.py", line 22, in extract_sources return tkp.steps.source_extraction.extract_sources(url, extraction_params) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/steps/source_extract
? ion.py", line 50, in extract_sources force_beam=extraction_params['force_beam'] File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/sourcefinder/image.py", line 401, in extract det * self.rmsmap, anl \ self.rmsmap, deblend_nthresh, force_beam, File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/utility/memoize.py", line 29, in call self.memo[instance] = self.funct(instance) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/sourcefinder/image.py", line 114, in _get_rm return self._interpolate(self.grids['rms'], roundup=True) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/utility/memoize.py", line 29, in call self.memo[instance] = self.funct(instance) File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/sourcefinder/image.py", line 92, in _grids return self.grids() File "/soft/trap/aartfaac/lib/python2.7/site-packages/tkp/sourcefinder/image.py", line 218, in grids assert(len(useful_chunk) == 1) AssertionError peeyush@struis:~/pipeline$
Output available in banana as dataset 3. (Not putting images on dropbox, as they are available on struis.

hsuyeep commented 9 years ago

Full logs in peeyush@struis:~/pipeline/testaartfaac/logs/2015-04-01T13:47:25

timstaley commented 9 years ago

Could be a bunch of things here, but as a first step I would suggest double checking how the observation pointing co-ordinates are stored. In PR https://github.com/transientskp/tkp/pull/357 I took a look at the supplied observation, and the pointing centre (as read by the AARTFAAC accessor Gijs submitted in that PR) was set to 0,0, so it's not surprising you're getting junk co-ordinate transforms.

hsuyeep commented 9 years ago

I checked in the .MS header, you're right that the pointing center is set to 0,0:

Shape: 1 columns by 1 rows
Info: {'readme': '', 'subType': '', 'type': 'Image'}Table keywords: {'logtable': 'Table: /home/peeyush/pipeline/testaartfaac/2013_images/F5.72174e+07_S0-0_T20-11-2013_09-58-07.image/logtable', 
<snip>
'pointingcenter': {'initial': True, 'value': array([ 0.,  0.])}
</snip>

There is , however, the AZEL specified coord. system, and there is a 'direction' keyword which points to zenith:

'direction': {'m1': {'unit': 'rad', 'value': 1.5707963267948966}, 'type': 'direction', 'm0': {'value': 0.0, 'unit': 'rad'}, 'refer': 'J2000'},

so the description seems complete. @Error323, I think we need to calculate the RA/DEC corresponding to the time of observation and put that in as the pointing center. Not sure that this is the cause of the assertion fail, since TraP seemed to work fine for the previous 40 or so images.

Full image header follows:

Table summary: /home/peeyush/pipeline/testaartfaac/2013_images/F5.72174e+07_S0-0_T20-11-2013_09-58-07.image
Shape: 1 columns by 1 rows
Info: {'readme': '', 'subType': '', 'type': 'Image'}
Table keywords: {'logtable': 'Table: /home/peeyush/pipeline/testaartfaac/2013_images/F5.72174e+07_S0-0_T20-11-2013_09-58-07.image/logtable', 'coords': {'spectral1': {'waveUnit': 'mm', 'wcs': {'pc': 1.0, 'crpix': 0.0, 'crval': 57217408.0, 'ctype': 'FREQ\x00\x00\x00\x00\x00', 'cdelt': 0.0}, 'name': 'Frequency', 'restfreqs': array([ 0.]), 'velUnit': 'km/s', 'system': 'LSRK', 'version': 2, 'velType': 0, 'restfreq': 0.0, 'formatUnit': '', 'nativeType': 0, 'unit': 'Hz', 'conversion': {'position': {'m1': {'value': 0.0, 'unit': 'rad'}, 'm2': {'unit': 'm', 'value': 0.0}, 'type': 'position', 'm0': {'value': 0.0, 'unit': 'rad'}, 'refer': 'ITRF'}, 'direction': {'m1': {'unit': 'rad', 'value': 1.5707963267948966}, 'type': 'direction', 'm0': {'value': 0.0, 'unit': 'rad'}, 'refer': 'J2000'}, 'epoch': {'type': 'epoch', 'm0': {'unit': 'd', 'value': 0.0}, 'refer': 'LAST'}, 'system': 'LSRK'}}, 'stokes2': {'stokes': ['I'], 'crpix': array([ 0.]), 'axes': ['Stokes'], 'cdelt': array([ 1.]), 'pc': array([[ 1.]]), 'crval': array([ 1.])}, 'pixelreplace2': array([ 0.]), 'telescope': 'AARTFAAC All Sky Monitor', 'observer': 'AARTFAAC Project', 'worldmap2': array([3], dtype=int32), 'pixelreplace1': array([ 0.]), 'pointingcenter': {'initial': True, 'value': array([ 0.,  0.])}, 'obsdate': {'type': 'epoch', 'm0': {'unit': 'd', 'value': 56616.41536326506}, 'refer': 'UTC'}, 'worldreplace1': array([ 57217408.]), 'worldreplace0': array([  4.50000000e+001,   1.58101007e-322]), 'worldmap1': array([2], dtype=int32), 'worldmap0': array([0, 1], dtype=int32), 'pixelmap2': array([3], dtype=int32), 'pixelmap1': array([2], dtype=int32), 'pixelmap0': array([0, 1], dtype=int32), 'direction0': {'crpix': array([ 512.,  512.]), 'longpole': 0.0, 'projection': 'SIN', 'projection_parameters': array([ 0.,  0.]), 'axes': ['Longitude', 'Latitude'], 'system': 'AZEL', 'cdelt': array([ 0.00195722,  0.00195722]), 'pc': array([[ 1.,  0.], [ 0.,  1.]]), 'conversionSystem': 'AZEL', 'crval': array([ 4.71238898,  1.57079633]), 'units': ['rad', 'rad'], 'latpole': 90.0}, 'telescopeposition': {'m1': {'value': 0.9203091326659768, 'unit': 'rad'}, 'm2': {'unit': 'm', 'value': 6364618.852280442}, 'type': 'position', 'm0': {'value': 0.1199012977060484, 'unit': 'rad'}, 'refer': 'ITRF'}, 'pixelreplace0': array([ 0.,  0.]), 'worldreplace2': array([ 1.])}, 'imageinfo': {'restoringbeam': {'major': {'unit': 'deg', 'value': 1.0}, 'positionangle': {'value': 0.0, 'unit': 'deg'}, 'minor': {'value': 1.0, 'unit': 'deg'}}, 'objectname': 'Aartfaac image', 'imagetype': 'Intensity'}}
Columns: ['map']

hsuyeep commented 9 years ago

So this is more for @Error323 : The pointing center seems to be set OK. The frequency is apparently not getting set. I get the following warning while running TraP: INFO:tkp.distribute.serial.tasks:running persistence task 2015-04-23 19:20:34 INFO FITSCoordinateUtil::toFITSHeader There is no axis with type DEC or LAT. Cannot identify latitude axis for WCS. Will assume axis 2 as default. 2015-04-23 19:20:34 SEVERE SpectralCoordinate::toFITS (file /build/buildd/casacore-1.7.0/coordinates/Coordinates/SpectralCoordinate.cc, line 1685) Zero or negative frequency. INFO:tkp.steps.persistence:Saved local copy of F5.72174e+07_S0-1_T20-11-2013_09-23-00.image on vlo.science.uva.nl INFO:tkp.steps.persistence:Extracting metadata from /home/peeyush/pipeline/testaartfaac/2013_images/F5.72174e+07_S0-1_T20-11-2013_09-23-00.image

Banana gets the pointing center and the bandwidth, but ofcourse not the frequency: Dataset TRAP dataset Centre 58.701, 52.915 start date (UT) Nov. 20, 2013, 10:23 a.m. Integration time 1.000 Tau None Filename F5.72174e+07_S0-1_T20-11-2013_09-23-00.image Rejections no Number of sources 5 Frequency Hz 0.0000 Assigned Band 0 MHz Bandwidth Hz 2.441e+05

gijzelaerr commented 9 years ago

@Error323 @hsuyeep has this been implemented in the AARTFAAC pipeline?

hsuyeep commented 9 years ago

The frequency metadata is now being correctly filled, but I think we need to test this a lot more due to the multiple issues that are contributing to this. Currently, AA images are not explicitlly crashing TraP (although the output is not correct). Recommend we first get teh source finder issues out of the way.

gijzelaerr commented 9 years ago

This has been fixed by filtering out imaging containing NaN values (calibration error)

timstaley commented 9 years ago

NaN-check merged in #452

transientskp / tkp

[AARTFAAC] TraP sourcefinder assertion fail on aartfaac-imaging-pipeline generated images #415