search

e-merlin / eMERLIN_CASA_pipeline

This is CASA eMERLIN pipeline to calibrate data from the e-MERLIN array. Please fork the repository before making any changes and read the Coding Practices page in the wiki. Please add issues with the pipeline in the issues tab.
GNU General Public License v3.0
14 stars 11 forks source link

AOflagger not running in some data sets #64

Closed jmoldon closed 6 years ago

jmoldon commented 6 years ago

AOflagger does not start complaining about non-existing row in ANTENNA table. The ANTENNA table has indeed 7 rows (0 Lo to 6 Cm).

The same problem when running aoflagger outside the pipeline, with or without fields or strategy parameters. A simple aoflagger CY3202_02_C_20160220.ms fails.

2017-10-20 10:35:25 | INFO | Running AOFLagger for field 1407+284 (3) using strategy CASA_eMERLIN_pipeline/aoflagger_strategies/default/1407+284.rfis
AOFlagger 2.9.0 (2016-12-20) command line application
This program will execute an RFI strategy as can be created with the RFI gui
and executes it on one or several observations.

Author: André Offringa (offringa@gmail.com)

Modified single-baseline strategy so it will execute strategy on all baselines and write flags.
Starting strategy on 2017-Oct-20 11:35:25.148382
0% : strategy...
0% : +-For each measurement set...
0% : +-+-Processing measurement set CY3202_02_C_20160220.ms...
terminate called after throwing an instance of 'casa::TableError'
  what():  SSMIndex::getIndex - access to non-existing row 7 in /mirror1/scratch/jmoldon/CY3202_pipe/CY3202_02_C_20160220/CY3202_02_C_20160220.ms/ANTENNA

casaplotms and listobs have no problems reading the data:

Antennas: 7:
  ID   Name  Station   Diam.    Long.         Lat.                Offset from array center (m)                ITRF Geocentric coordinates (m)        
                                                                     East         North     Elevation               x               y               z
  0    Lo    e-MERLIN:0176.0 m   -002.18.25.8  +53.03.07.9      19400.8559    21183.6803     6369.1778  3822252.643000  -153995.683000  5086051.443000
  1    Mk2   e-MERLIN:0224.0 m   -002.18.08.9  +53.02.58.7      19713.9103    20897.1596     6334.4681  3822473.365000  -153692.318000  5085851.303000
  2    Kn    e-MERLIN:0525.0 m   -002.59.44.9  +52.36.18.4     -26733.5549   -28428.6831     6480.6814  3859711.503000  -201995.077000  5056134.251000
  3    De    e-MERLIN:0625.0 m   -002.08.35.0  +51.54.50.9      30394.6148  -105100.8391     6688.6339  3923069.171000  -146804.368000  5009320.528000
  4    Pi    e-MERLIN:0725.0 m   -002.26.38.3  +53.06.16.2      10235.7831    26985.6080     6271.3637  3817176.561000  -162921.179000  5089462.057000
  5    Da    e-MERLIN:0825.0 m   -002.32.03.3  +52.58.18.5       4186.5058    12262.8833     6330.1699  3828714.513000  -169458.995000  5080647.749000
  6    Cm    e-MERLIN:0932.0 m   +000.02.19.5  +51.58.50.2     176561.6720   -97724.9405     6660.8614  3919982.752000     2651.982000  5013849.826000

I have seen this same problem with other data sets.

jmoldon commented 6 years ago

I found the same problem for another dataset. For project CY3213_01_L_20160318. In this case we have 6 antennas. No problems with CASA processing the data, but aoflagger complains about a non-existing row 6.

For this data set, individual sources were exported using data_finder in emproc1. The same problem appears when trying to flag the concatenated dataset, or just a single source exported from an individual FITS-IDI.


Antennas: 6:
  ID   Name  Station   Diam.    Long.         Lat.                Offset from array center (m)                ITRF Geocentric coordinates (m)        
                                                                     East         North     Elevation               x               y               z
  0    Mk2   e-MERLIN:0224.0 m   -002.18.08.9  +53.02.58.7      19713.9103    20897.1596     6334.4681  3822473.365000  -153692.318000  5085851.303000
  1    Kn    e-MERLIN:0525.0 m   -002.59.44.9  +52.36.18.4     -26733.5549   -28428.6831     6480.6814  3859711.503000  -201995.077000  5056134.251000
  2    De    e-MERLIN:0625.0 m   -002.08.35.0  +51.54.50.9      30394.6148  -105100.8391     6688.6339  3923069.171000  -146804.368000  5009320.528000
  3    Pi    e-MERLIN:0725.0 m   -002.26.38.3  +53.06.16.2      10235.7831    26985.6080     6271.3637  3817176.561000  -162921.179000  5089462.057000
  4    Da    e-MERLIN:0825.0 m   -002.32.03.3  +52.58.18.5       4186.5058    12262.8833     6330.1699  3828714.513000  -169458.995000  5080647.749000
  5    Cm    e-MERLIN:0932.0 m   +000.02.19.5  +51.58.50.2     176561.6720   -97724.9405     6660.8614  3919982.752000     2651.982000  5013849.826000
AOFlagger 2.9.0 (2016-12-20) command line application
This program will execute an RFI strategy as can be created with the RFI gui
and executes it on one or several observations.

Author: André Offringa (offringa@gmail.com)

Modified single-baseline strategy so it will execute strategy on all baselines and write flags.
Starting strategy on 2017-Nov-13 13:51:16.573500
0% : strategy...
0% : +-For each measurement set...
0% : +-+-Processing measurement set CY3213_01_L_20160318_avg.ms...
terminate called after throwing an instance of 'casa::TableError'
  what():  SSMIndex::getIndex - access to non-existing row 6 in /mirror1/scratch/jmoldon/emerlin/CY3213/CY3213_01_L_20160318/CY3213_01_L_20160318_avg.ms/ANTENNA
Aborted (core dumped)
jmoldon commented 6 years ago

Following Andre's advice:

I think what might be the issue is that Casacore was upgraded and/or recompiled on your machine while aoflagger was not recompiled. This typically creates such weird Casacore errors that don't make sense. If that's the case, it might be enough to remove your aoflagger build directory and recompile it.

So that is how I solved it: just reinstalled casacore and aoflagger. I have the versions from anaconda.

We don't know the origin or the problem, but that is the solution.