search

e-merlin / eMERLIN_CASA_pipeline

This is CASA eMERLIN pipeline to calibrate data from the e-MERLIN array. Please fork the repository before making any changes and read the Coding Practices page in the wiki. Please add issues with the pipeline in the issues tab.
GNU General Public License v3.0
14 stars 11 forks source link

importfitsidi has divided a dataset in too many scans #85

Closed jmoldon closed 6 years ago

jmoldon commented 6 years ago

Run with normal parameters:

2017-11-14 09:56:39 | INFO | Pipeline version: v0.6.7
2017-11-14 09:56:39 | INFO | Using github branch: master
2017-11-14 09:56:39 | INFO | github last commit: 9e7c9ae

The resulting MS has hundreds of scans, divided every 97 seconds. Subset of listobs:

              17:44:32.5 - 17:46:09.5    42      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:46:09.5 - 17:47:46.5    43      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:47:46.5 - 17:49:23.5    44      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:49:23.5 - 17:51:00.5    45      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:51:00.5 - 17:52:37.5    46      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:52:37.5 - 17:54:14.5    47      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:54:14.5 - 17:55:51.5    48      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:55:51.5 - 17:57:28.5    49      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:57:28.5 - 17:59:05.5    50      3 0319+415                 16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              17:59:05.5 - 17:59:58.5    51      3 0319+415                  8904  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:00:01.5 - 18:01:36.5    52      4 2007+4029                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:01:36.5 - 18:02:58.5    53      4 2007+4029                13776  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:03:01.5 - 18:04:36.5    54      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:04:36.5 - 18:06:13.5    55      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:06:13.5 - 18:07:50.5    56      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:07:50.5 - 18:09:27.5    57      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:09:27.5 - 18:09:58.5    58      2 2031+4131                 5208  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:10:01.5 - 18:11:36.5    59      4 2007+4029                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:11:36.5 - 18:12:58.5    60      4 2007+4029                13776  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:13:01.5 - 18:14:36.5    61      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:14:36.5 - 18:16:13.5    62      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:16:13.5 - 18:17:50.5    63      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:17:50.5 - 18:19:27.5    64      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:19:27.5 - 18:19:58.5    65      2 2031+4131                 5208  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:20:01.5 - 18:21:36.5    66      4 2007+4029                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:21:36.5 - 18:22:58.5    67      4 2007+4029                13776  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:23:01.5 - 18:24:36.5    68      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:24:36.5 - 18:26:13.5    69      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:26:13.5 - 18:27:50.5    70      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:27:50.5 - 18:29:27.5    71      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:29:27.5 - 18:29:58.5    72      2 2031+4131                 5208  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:30:01.5 - 18:31:36.5    73      4 2007+4029                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:31:36.5 - 18:32:58.5    74      4 2007+4029                13776  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:33:01.5 - 18:34:36.5    75      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:34:36.5 - 18:36:13.5    76      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:36:13.5 - 18:37:50.5    77      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:37:50.5 - 18:39:27.5    78      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:39:27.5 - 18:39:58.5    79      2 2031+4131                 5208  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:40:01.5 - 18:41:36.5    80      4 2007+4029                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:41:36.5 - 18:42:58.5    81      4 2007+4029                13776  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:43:01.5 - 18:44:36.5    82      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979, 0.979] 
              18:44:36.5 - 18:46:13.5    83      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:46:13.5 - 18:47:50.5    84      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:47:50.5 - 18:49:27.5    85      2 2031+4131                16296  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1] 
              18:49:27.5 - 18:49:58.5    86      2 2031+4131                 5208  [0,1,2,3,4,5,6,7]  [1, 1, 1, 1, 1, 1, 1, 1]

CASA command executed:

 importfitsidi(fitsidifile=['./DATA/DD5007_01_L_20171109.fits'],vis="DD5007_01_L_20171109.ms",constobsid=True,scanreindexgap_s=15.0)

Data was exported to a single FITS-IDI file with data_tool from the command line.

jmoldon commented 6 years ago

I think it was an external problem caused by manually killing importfitsidi just before it finished. I repeated the step and it is OK.

pahjbo commented 6 years ago

that probably reflects the actual structure of the fits IDI - the UV Data tables are kept reasonably small for performance reasons