search

lofar-astron / DP3

DP3: streaming processing pipeline for radio interferometric data
GNU General Public License v3.0
15 stars 10 forks source link

Different behaviour of predict gaincal in DPPP 4.1 and 5.1.0 #329

Open adrabent opened 3 years ago

adrabent commented 3 years ago

Dear all,

I have a regular TGSS skymodel and want to run DPPP gaincal on it. I created a sourcedb file with

makesourcedb in=target.skymodel out=target.sourcedb format='<' outtype=blob

and I get the following response:

makesourcedb: Version GvD 2013-May-16
Wrote 1 patches (out of 1) and 195 sources (out of 195) into target.sourcedb

If I know run the following calibration parset with DPPP 4.1:

steps                                    =   [filter,gaincal]
filter.type                              =   filter
filter.blrange                           =   [150, 999999]
gaincal.type                             =   gaincal
gaincal.parmdb                           =   instrument_target.h5
gaincal.caltype                          =   phaseonly
gaincal.sourcedb                         =   target.sourcedb
gaincal.maxiter                          =   50
gaincal.solint                           =   1
gaincal.nchan                            =   0
gaincal.tolerance                        =   1e-3
gaincal.propagatesolutions               =   True
gaincal.usebeammodel                     =   True
gaincal.usechannelfreq                   =   True
gaincal.beammode                         =   array_factor
gaincal.onebeamperpatch                  =   False

I get the following (expected) output:

MSReader
  input MS:       /home/alex/test/L228163_SB000_uv.dppp.MS
  band            0
  startchan:      0  (0)
  nchan:          4  (0)
  ncorrelations:  4
  nbaselines:     1830
  first time:     2014/05/08/19:11:03
  last time:      2014/05/09/04:49:01
  ntimes:         6891
  time interval:  5.03316
  DATA column:    DATA
  WEIGHT column:  WEIGHT_SPECTRUM
  autoweight:     false
Filter filter.
  startchan:      0  (0)
  nchan:          4  (0)
  Baseline selection:
    baseline:
    corrtype:
    blrange:      [150,999999]
  remove:         0
GainCal gaincal.
  H5Parm:              instrument_target.h5
  solint:              1
  nchan:               4
  max iter:            50
  tolerance:           0.001
  caltype:             phaseonly
  apply solution:      false
  propagate solutions: true
  detect stalling:     true
  use model column:    false
  Baseline selection:
    baseline:
    corrtype:
    blrange:      []
Predict gaincal.
  sourcedb:           target.sourcedb
   number of patches: 195
   number of sources: 195
   all unpolarized:   false
  apply beam:         true
   mode:              array_factor
   use channelfreq:   true
   one beam per patch:false
  operation:          replace
  threads:            56
MSUpdater msout.
  MS:             /home/alex/test/L228163_SB000_uv.dppp.MS
  datacolumn:     DATA 
  weightcolumn    WEIGHT_SPECTRUM
  writing:        data flags
  Compressed:     no   

  flush:          0

Processing 6891 time slots ...

0%....10....20....30....40....50....60....70....80....90....100%
Finishing processing ...

NaN/infinite data flagged in reader
===================================

Percentage of flagged visibilities detected per correlation:
  [0,0,0,0] out of 50442120 visibilities   [0%, 0%, 0%, 0%]
0 missing time slots were inserted

Total NDPPP time     318.38 real     2987.08 user      356.54 system
    5.0% MSReader
    1.1% Filter filter.
   84.7% GainCal gaincal.
           58.6% of it spent in predict
           33.0% of it spent in reordering visibility data
            8.4% of it spent in estimating gains and computing residuals
            0.0% of it spent in writing gain solutions to disk
        Converged: 6857, stalled: 34, non converged: 0, failed: 0
        Iters converged: 3, stalled: 10, non converged: 0, failed: 0
    8.2% MSUpdater msout.

If I do the same just using DPPP 5.1.0 I get a different amount of patches than sources, and sources are suddenly treated as all unpolarized:

MSReader
  input MS:       /home/alex/test/L228163_SB000_uv.dppp.MS
  band            0
  startchan:      0  (0)
  nchan:          4  (0)
  ncorrelations:  4
  nbaselines:     1830
  first time:     2014/05/08/19:11:03
  last time:      2014/05/09/04:49:01
  ntimes:         6891
  time interval:  5.03316
  DATA column:    DATA
  WEIGHT column:  WEIGHT_SPECTRUM
  autoweight:     false
Filter filter.
  startchan:      0  (0)
  nchan:          4  (0)
  Baseline selection:
    baseline:
    corrtype:
    blrange:      [150,999999]
  remove:         0
GainCal gaincal.
  H5Parm:              instrument_target.h5
  solint:              1
  nchan:               4
  max iter:            50
  tolerance:           0.001  
  caltype:             diagonalphase
  apply solution:      false  
  propagate solutions: true   
  detect stalling:     true   
  use model column:    false  
  Baseline selection:
    baseline:
    corrtype:
    blrange:      []
Predict gaincal.
  sourcedb:           target.sourcedb
   number of patches: 189
   number of sources: 195
   all unpolarized:   true
  apply beam:         true
   mode:              array_factor
   use channelfreq:   true
   one beam per patch:false   
   beam proximity lim:0.000290888
  operation:          replace 
  threads:            56
MSUpdater msout.
  MS:             /home/alex/test/L228163_SB000_uv.dppp.MS
  datacolumn:     DATA
  weightcolumn    WEIGHT_SPECTRUM
  writing:        data flags  
  Compressed:     no

  flush:          0

Processing 6891 time slots ...

0%....10....20....30....40....50....60....70....80....90....100%
Finishing processing ...

NaN/infinite data flagged in reader
===================================

Percentage of flagged visibilities detected per correlation:
  [0,0,0,0] out of 50442120 visibilities   [0%, 0%, 0%, 0%]
0 missing time slots were inserted

Total NDPPP time     542.42 real     4663.57 user      264.58 system
    2.6% MSReader
    0.6% Filter filter.
   92.3% GainCal gaincal.
           35.6% of it spent in predict
           17.4% of it spent in reordering visibility data
           46.9% of it spent in estimating gains and computing residuals
            0.0% of it spent in writing gain solutions to disk
        Converged: 6857, stalled: 34, non converged: 0, failed: 0
        Iters converged: 3, stalled: 10, non converged: 0, failed: 0
    4.4% MSUpdater msout.

The good news is that I don't see any difference in the calibration results in my case, but this does not mean that there could be one in principle if sources and patches are somewhat treated differently, or if sources are considered to be polarized/unpolarized. The used skymodel is attached.

Cheers, Alex

target.skymodel.gz

target.skymodel.gz