search

lofar-astron / PiLL

Pipeline for Lofar LBA
5 stars 4 forks source link

"Channels are not evenly spaced"? #21

Closed mknapp55 closed 7 years ago

mknapp55 commented 7 years ago

I got the following error at step selfcal-parmdbFRtar:

FATAL - bbs-reducer terminated due to an exception: [AssertError: Assertion: allNearAbs(upFreq-lowFreq,freqstep0,1.e3); Channels are not evenly spaced. This is not supported.]

I'm working with Cycle 0 data that has 120 SB for cal and target instead of the now-standard 244. All of the SB are present for both target and calibrator. I re-preprocessed the raw data so that it has 5 second integration time and 8 channels/SB. My parset averages to 4 ch/SB. What's causing this error?

I'm not using the latest update of PiLL (thanks for that, btw!). Has anything changed in the update that might fix this? I have four more datasets just like this that I'd like to process...

revoltek commented 7 years ago

maybe obvious but: are the subbands evenly spaced beside being all present? Is this problem in the cal or selfcal part of the pipeline?

mknapp55 commented 7 years ago

@revoltek Yes, they are evenly spaced as far as I can tell (and I checked both the SB spacing and the channel spacing). This error occurred in selfcal-parmdbFRtar, but the equivalent cal-parmdbFRtar on the calibrator didn't fail.

The SBs are 390.625 kHz apart (REF_FREQ SB1 to REF_FREQ SB1) and channels are 24.4140625 kHz wide. Each SB is 170.8984375 kHz wide (which makes sense since the cal/targ SB alternate and a few edge channels are knocked off in preprocessing).

Previously, I ran the pipeline with one target SB missing. I fixed that (and triple-checked that all SB were present) and reran after deleting all of the target MSs in the working directory and resetting the statefile to the beginning of the time_split step. The formerly missing SB is present in the working directory (as one of the L*.time_split-flagtar files), so it was picked up in the rerun.

adrabent commented 7 years ago

Could you try to run the BBS task outside the pipeline? The corresponding parset is located in ./parsets/selfcal-parmdbFRtar.parset. You may have to specify the observations, some options and the skymodel in the call:

calibrate-stand-alone -v -f -n [your-TC].time_split-split ./parsets/selfcal-parmdbFRtar.parset /path/to/skymodel/skymodel

Please repeat this task on the target data actually BEFORE it has been split into time chunks and/or BEFORE it was concatenated. Maybe we can figure out at which step the problem is induced.

mknapp55 commented 7 years ago

The problem apparently happens at the concatenation step. See results below.

Copied, flagged target MSs:

mknapp@jvla:/data/scripts/working_L85569/pipeline$ calibrate-stand-alone -v -f -n L85569_SAP000_SB092_uv.preproc2.time_split-flagtar ./parsets/selfcal-parmdbFRtar.parset L85569_SAP000_SB000_uv.preproc2.MS.sky
calibrate-stand-alone (top-level calibration script, stand-alone version) Tue Jul 4 13:39:41 PDT 2017

arguments:
    observation: /data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar
    parset: /data/scripts/working_L85569/pipeline/parsets/selfcal-parmdbFRtar.parset
    catalog: /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2.MS.sky

    verbose: 1
    force: 1
    add_columns: 0
    dry: 0
    sourcedb: 
    sourcedb name: sky
    replace sourcedb: 1
    parmdb: 
    parmdb name: instrument
    replace parmdb: 1
    number of threads: 1

Tue Jul  4 13:39:41 PDT 2017 create sourcedb
rm -rf "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar/sky" && makesourcedb in="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2.MS.sky" out="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar/sky" format='<'
log4cplus:WARN Property configuration file "makesourcedb.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
makesourcedb: Version GvD 2013-May-16
No format string found; using default format
Wrote 0 patches (out of 0) and 0 sources (out of 0) into /data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar/sky
Tue Jul  4 13:39:42 PDT 2017 create parmdb
rm -rf "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar/instrument" && parmdbm << [default parmdb commands]
log4cplus:WARN Property configuration file "parmdbm.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
Command: Command: Gain:0:0:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:0:0:Ampl
Command: Gain:1:1:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:1:1:Ampl
Command: Gain:0:0:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:0:0:Real
Command: Gain:1:1:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:1:1:Real
Command: DirectionalGain:0:0:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:0:0:Ampl
Command: DirectionalGain:1:1:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:1:1:Ampl
Command: DirectionalGain:0:0:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:0:0:Real
Command: DirectionalGain:1:1:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:1:1:Real
Command: Clock  type=scalar
    values:  0  shape=[1, 1]
    pert=1e-15 pert_rel=y
Wrote new defaultvalue record for parameter Clock
Command: 
Tue Jul  4 13:39:42 PDT 2017 execute bbs-reducer
bbs-reducer --numthreads=1 --sourcedb="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar/sky" --parmdb="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar/instrument" "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB092_uv.preproc2.time_split-flagtar" "/data/scripts/working_L85569/pipeline/parsets/selfcal-parmdbFRtar.parset"
log4cplus:WARN Property configuration file "bbs-reducer.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
INFO - bbs-reducer: version = Unknown (in CMakeLists.txt: 1.0)
 branch = 
 overall revision  = Unknown
 package revision  = Unknown (last change in package)
 built on lgw01-13 by buildd at Mar 22 2017 08:50:14
  Unknown files were different from the repository
 packages used: 
  BBSKernel-Unknown (rev.Unknown) Unknown changed files
  PLC-Unknown (rev.Unknown) Unknown changed files
  ParmDB-Unknown (rev.Unknown) Unknown changed files
  StationResponse-Unknown (rev.Unknown) Unknown changed files
  Transport-Unknown (rev.Unknown) Unknown changed files
  LMWCommon-Unknown (rev.Unknown) Unknown changed files
  ElementResponse-Unknown (rev.Unknown) Unknown changed files
  Blob-Unknown (rev.Unknown) Unknown changed files
  Common-Unknown (rev.Unknown) Unknown changed files

DEBUG - Measurement 0 contains 2717820 row(s).
DEBUG - Measurement dimensions:
Frequency     : 65.9 MHz - 66.1 MHz
Bandwidth     : 0.195 MHz (4 channel(s) of 4.88e+04 Hz)
Time          : 2013/01/24/21:37:11 - 2013/01/25/03:37:11
Duration      : 6 hour (4314 sample(s) of 5.01 s on average)
Baseline count: 630
Correlation(s): [ XX XY YX YY ]
WARN - 
*** WARNING: the following parset keywords were not used ***
             maybe they are misspelled
    ["catalog","force","numthreads","observation","parmdb-name"]

INFO - Selected time range (sample): [0,4313] (out of: 4314)
INFO - Chunk size (sample): 30
DEBUG - Executing a NextChunk command:
Command: NextChunk
Frequency: 65.9 - 66.1 MHz
Time: 2013/01/24/21:37:11 - 2013/01/24/21:39:41

INFO - Starting chunk 1 of 143 (0%)
DEBUG - Reading chunk from column: DATA
DEBUG - Selection contains 595 baseline(s), 30 timeslot(s), 4 channel(s), and 4 correlation(s).
DEBUG - Buffer size: 13.3461 MB.
DEBUG - Reading visibilities from column: DATA
DEBUG - Reading noise covariance from linked column: WEIGHT_SPECTRUM
DEBUG - Read time: timer: avg = 7.12 ms, total =  214 ms, count =        30
DEBUG - Copy time: timer: avg = 3.07 ms, total =   92 ms, count =        30
DEBUG - Chunk dimensions: 
Frequency     : 65.9 MHz - 66.1 MHz
Bandwidth     : 0.195 MHz (4 channel(s) of 4.88e+04 Hz)
Time          : 2013/01/24/21:37:11 - 2013/01/24/21:39:41
Duration      : 0.0417 hour (30 sample(s) of 5.01 s on average)
Baseline count: 595
Correlation(s): [ XX XY YX YY ]
DEBUG - Executing a Solve command:
Step: Solve
. Name: solve
. Full name: solve
. Baselines: *&
. Correlations: []
. Model configuration:
. . Bandpass enabled: false
. . Clock enabled: false
. . Gain enabled: false
. . TEC enabled: false
. . Common rotation enabled: false
. . Common scalar phase enabled: false
. . Direction dependent gain enabled: false
. . Elevation cut enabled: false
. . Beam enabled: false
. . Direction dependent TEC enabled: false
. . Faraday rotation enabled: true
. . Polarization rotation enabled: false
. . Scalar phase enabled: false
. . Ionosphere enabled: false
. . Flagger enabled: false
. . Cache enabled: true
. . Sources: [@MODEL_DATA]
. Output column: 
. Write flags: true
. Write covariance: false
. Solve: 
. . Mode: COMPLEX
. . Algorithm: L2
. . L1 epsilon values: [0.0001,1e-05,1e-06]
. . Solvable parameters: ["RotationMeasure:*"]
. . Excluded parameters: []
. . Calibration groups: []
. . Cell size:
. . . Frequency (channels): 0
. . . Time (timestamps): 30
. . Cell chunk size: 30
. . Propagate solutions: false
. . Log:
. . . Name: solver_log
. . . Level: NONE
. . Flag on UV interval: false
. . Outlier rejection: false
. . Resample observed data: false
. . Phase shift observed data: false
. . Solver options:
. . . Max nr. of iterations: 1
. . . Epsilon value: 1e-08
. . . Epsilon derivative: 1e-08
. . . Colinearity factor: 1e-06
. . . LM factor: 0.001
. . . Balanced equations: false
. . . Use SVD: false
DEBUG - Unable to access the keywords of column: MODEL_DATA
FATAL - bbs-reducer terminated due to an exception: [BBSKernelException: Unable to access the keywords of column: MODEL_DATA]
in function std::__cxx11::string LOFAR::BBS::MeasurementAIPS::getLinkedCovarianceColumn(const string&, const string&) const
(/build/lofar-b_IP79/lofar-2.20.2/CEP/Calibration/BBSKernel/src/MeasurementAIPS.cc:1195)
Backtrace follows:
#0  0x7f6d48694f2c in ?? at ??:0
#1  0x7f6d48696e0b in ?? at ??:0
#2  0x7f6d48aaa071 in ?? at ??:0
#3  0x7f6d48aae52b in ?? at ??:0
#4  0x7f6d48ae65e7 in ?? at ??:0
#5  0x423a6e in ?? at ??:0
#6  0x42158a in ?? at ??:0
#7  0x7f6d46a0f830 in ?? at ??:0
#8  0x421f79 in ?? at ??:0

Concatenated MS:

mknapp@jvla:/data/scripts/working_L85569/pipeline$ calibrate-stand-alone -v -f -n L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat ./parsets/selfcal-parmdbFRtar.parset L85569_SAP000_SB000_uv.preproc2.MS.sky
calibrate-stand-alone (top-level calibration script, stand-alone version) Tue Jul 4 13:41:42 PDT 2017

arguments:
    observation: /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat
    parset: /data/scripts/working_L85569/pipeline/parsets/selfcal-parmdbFRtar.parset
    catalog: /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2.MS.sky

    verbose: 1
    force: 1
    add_columns: 0
    dry: 0
    sourcedb: 
    sourcedb name: sky
    replace sourcedb: 1
    parmdb: 
    parmdb name: instrument
    replace parmdb: 1
    number of threads: 1

Tue Jul  4 13:41:42 PDT 2017 create sourcedb
rm -rf "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat/sky" && makesourcedb in="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2.MS.sky" out="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat/sky" format='<'
log4cplus:WARN Property configuration file "makesourcedb.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
makesourcedb: Version GvD 2013-May-16
No format string found; using default format
Wrote 0 patches (out of 0) and 0 sources (out of 0) into /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat/sky
Tue Jul  4 13:41:43 PDT 2017 create parmdb
rm -rf "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat/instrument" && parmdbm << [default parmdb commands]
log4cplus:WARN Property configuration file "parmdbm.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
Command: Command: Gain:0:0:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:0:0:Ampl
Command: Gain:1:1:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:1:1:Ampl
Command: Gain:0:0:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:0:0:Real
Command: Gain:1:1:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:1:1:Real
Command: DirectionalGain:0:0:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:0:0:Ampl
Command: DirectionalGain:1:1:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:1:1:Ampl
Command: DirectionalGain:0:0:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:0:0:Real
Command: DirectionalGain:1:1:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:1:1:Real
Command: Clock  type=scalar
    values:  0  shape=[1, 1]
    pert=1e-15 pert_rel=y
Wrote new defaultvalue record for parameter Clock
Command: 
Tue Jul  4 13:41:43 PDT 2017 execute bbs-reducer
bbs-reducer --numthreads=1 --sourcedb="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat/sky" --parmdb="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat/instrument" "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_55MHz.msdpppconcat" "/data/scripts/working_L85569/pipeline/parsets/selfcal-parmdbFRtar.parset"
log4cplus:WARN Property configuration file "bbs-reducer.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
INFO - bbs-reducer: version = Unknown (in CMakeLists.txt: 1.0)
 branch = 
 overall revision  = Unknown
 package revision  = Unknown (last change in package)
 built on lgw01-13 by buildd at Mar 22 2017 08:50:14
  Unknown files were different from the repository
 packages used: 
  BBSKernel-Unknown (rev.Unknown) Unknown changed files
  PLC-Unknown (rev.Unknown) Unknown changed files
  ParmDB-Unknown (rev.Unknown) Unknown changed files
  StationResponse-Unknown (rev.Unknown) Unknown changed files
  Transport-Unknown (rev.Unknown) Unknown changed files
  LMWCommon-Unknown (rev.Unknown) Unknown changed files
  ElementResponse-Unknown (rev.Unknown) Unknown changed files
  Blob-Unknown (rev.Unknown) Unknown changed files
  Common-Unknown (rev.Unknown) Unknown changed files

DEBUG - Assertion: allNearAbs(upFreq-lowFreq,freqstep0,1.e3); Channels are not evenly spaced. This is not supported.
FATAL - bbs-reducer terminated due to an exception: [AssertError: Assertion: allNearAbs(upFreq-lowFreq,freqstep0,1.e3); Channels are not evenly spaced. This is not supported.]
in function void LOFAR::BBS::MeasurementAIPS::initDimensions()
(/build/lofar-b_IP79/lofar-2.20.2/CEP/Calibration/BBSKernel/src/MeasurementAIPS.cc:731)
Backtrace follows:
#0  0x7fe6f4ad504c in ?? at ??:0
#1  0x7fe6f4adb7a6 in ?? at ??:0
#2  0x4223c6 in ?? at ??:0
#3  0x42158a in ?? at ??:0
#4  0x7fe6f2e59830 in ?? at ??:0
#5  0x421f79 in ?? at ??:0

Time split MS:

mknapp@jvla:/data/scripts/working_L85569/pipeline$ calibrate-stand-alone -v -f -n L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split ./parsets/selfcal-parmdbFRtar.parset L85569_SAP000_SB000_uv.preproc2.MS.sky
calibrate-stand-alone (top-level calibration script, stand-alone version) Tue Jul 4 13:42:47 PDT 2017

arguments:
    observation: /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split
    parset: /data/scripts/working_L85569/pipeline/parsets/selfcal-parmdbFRtar.parset
    catalog: /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2.MS.sky

    verbose: 1
    force: 1
    add_columns: 0
    dry: 0
    sourcedb: 
    sourcedb name: sky
    replace sourcedb: 1
    parmdb: 
    parmdb name: instrument
    replace parmdb: 1
    number of threads: 1

Tue Jul  4 13:42:47 PDT 2017 create sourcedb
rm -rf "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split/sky" && makesourcedb in="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2.MS.sky" out="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split/sky" format='<'
log4cplus:WARN Property configuration file "makesourcedb.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
makesourcedb: Version GvD 2013-May-16
No format string found; using default format
Wrote 0 patches (out of 0) and 0 sources (out of 0) into /data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split/sky
Tue Jul  4 13:42:47 PDT 2017 create parmdb
rm -rf "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split/instrument" && parmdbm << [default parmdb commands]
log4cplus:WARN Property configuration file "parmdbm.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
Command: Command: Gain:0:0:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:0:0:Ampl
Command: Gain:1:1:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:1:1:Ampl
Command: Gain:0:0:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:0:0:Real
Command: Gain:1:1:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter Gain:1:1:Real
Command: DirectionalGain:0:0:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:0:0:Ampl
Command: DirectionalGain:1:1:Ampl  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:1:1:Ampl
Command: DirectionalGain:0:0:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:0:0:Real
Command: DirectionalGain:1:1:Real  type=scalar
    values:  1  shape=[1, 1]
    pert=1e-06 pert_rel=y
Wrote new defaultvalue record for parameter DirectionalGain:1:1:Real
Command: Clock  type=scalar
    values:  0  shape=[1, 1]
    pert=1e-15 pert_rel=y
Wrote new defaultvalue record for parameter Clock
Command: 
Tue Jul  4 13:42:47 PDT 2017 execute bbs-reducer
bbs-reducer --numthreads=1 --sourcedb="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split/sky" --parmdb="/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split/instrument" "/data/scripts/working_L85569/pipeline/L85569_SAP000_SB000_uv.preproc2_12205EA0At_63MHz_TC4.time_split-split" "/data/scripts/working_L85569/pipeline/parsets/selfcal-parmdbFRtar.parset"
log4cplus:WARN Property configuration file "bbs-reducer.log_prop" not found.
log4cplus:WARN Using basic logging configuration.
INFO - bbs-reducer: version = Unknown (in CMakeLists.txt: 1.0)
 branch = 
 overall revision  = Unknown
 package revision  = Unknown (last change in package)
 built on lgw01-13 by buildd at Mar 22 2017 08:50:14
  Unknown files were different from the repository
 packages used: 
  BBSKernel-Unknown (rev.Unknown) Unknown changed files
  PLC-Unknown (rev.Unknown) Unknown changed files
  ParmDB-Unknown (rev.Unknown) Unknown changed files
  StationResponse-Unknown (rev.Unknown) Unknown changed files
  Transport-Unknown (rev.Unknown) Unknown changed files
  LMWCommon-Unknown (rev.Unknown) Unknown changed files
  ElementResponse-Unknown (rev.Unknown) Unknown changed files
  Blob-Unknown (rev.Unknown) Unknown changed files
  Common-Unknown (rev.Unknown) Unknown changed files

DEBUG - Assertion: allNearAbs(upFreq-lowFreq,freqstep0,1.e3); Channels are not evenly spaced. This is not supported.
FATAL - bbs-reducer terminated due to an exception: [AssertError: Assertion: allNearAbs(upFreq-lowFreq,freqstep0,1.e3); Channels are not evenly spaced. This is not supported.]
in function void LOFAR::BBS::MeasurementAIPS::initDimensions()
(/build/lofar-b_IP79/lofar-2.20.2/CEP/Calibration/BBSKernel/src/MeasurementAIPS.cc:731)
Backtrace follows:
#0  0x7fe4b1da504c in ?? at ??:0
#1  0x7fe4b1dab7a6 in ?? at ??:0
#2  0x4223c6 in ?? at ??:0
#3  0x42158a in ?? at ??:0
#4  0x7fe4b0129830 in ?? at ??:0
#5  0x421f79 in ?? at ??:0
adrabent commented 7 years ago

Is the amount of channels before and after concat the same or is this step adding channels?

AHorneffer commented 7 years ago

The SBs are 390.625 kHz apart (REF_FREQ SB1 to REF_FREQ SB1) and channels are 24.4140625 kHz wide. Each SB is 170.8984375 kHz wide (which makes sense since the cal/targ SB alternate and a few edge channels are knocked off in preprocessing).

When you kick out the edges of a subband - instead of only flagging the channels - then you cannot concatenate these subband in frequency, keep more than one channel per subband, and have the data usable with LOFAR software (BBS and (AFAIK sill) NDPPP). If you think about it: the step from one channel inside a subband to the next is the channel-width, but the step from the last channel in one SB to the first channel in the next SB is the channel-width plus the gap between the two subbands.

adrabent commented 7 years ago

@AHorneffer PiLL also uses the sort_times_into_freqGroups.py script and mapfile modifications as in prefactor. Shouldn't this script add missing "subbands/channels", so that the outfile is enforced to be evenly spaced?

AHorneffer commented 7 years ago

@adrabent Nope! That script only fills in missing subbands. (Well, that script prepares the call to NDPPP so that NDPPP fills in missing subbands.)

Filling in gaps between subbands is much harder, not implemented in NDPPP, and also not always possible. (It would theoretically be possible here, because the gap between two subbands is by chance as wide as one channel inside a subband. But I don't know of any software that can do this.)

mknapp55 commented 7 years ago

@adrabent The concat step does change the number of channels - it goes from 4 per SB to 40 since I set chunks = 10 in the parset. Is that the expected behavior? See below for the SB/channel setup and widths for the three steps. L*SB001.time_split-flagtar:

In [15]: t.getcol('REF_FREQUENCY')
Out[15]: array([ 30468750.])

In [16]: t.getcol('TOTAL_BANDWIDTH')
Out[16]: array([ 195312.5])

In [17]: t.getcol('NUM_CHAN')
Out[17]: array([4], dtype=int32)

In [18]: t.getcol('CHAN_FREQ')
Out[18]: 
array([[ 30393981.93359375,  30442810.05859375,  30491638.18359375,
         30540466.30859375]])

In [19]: t.getcol('CHAN_WIDTH')
Out[19]: array([[ 48828.125,  48828.125,  48828.125,  48828.125]])

L*31MHz.msdpppconcat:

In [21]: t2.getcol('REF_FREQUENCY')
Out[21]: array([ 31834411.62109375])

In [22]: t2.getcol('TOTAL_BANDWIDTH')
Out[22]: array([ 1953125.])

In [23]: t2.getcol('NUM_CHAN')
Out[23]: array([40], dtype=int32)

In [24]: t2.getcol('CHAN_FREQ')
Out[24]: 
array([[ 30003356.93359375,  30052185.05859375,  30101013.18359375,
         30149841.30859375,  30393981.93359375,  30442810.05859375,
         30491638.18359375,  30540466.30859375,  30784606.93359375,
         30833435.05859375,  30882263.18359375,  30931091.30859375,
         31175231.93359375,  31224060.05859375,  31272888.18359375,
         31321716.30859375,  31565856.93359375,  31614685.05859375,
         31663513.18359375,  31712341.30859375,  31956481.93359375,
         32005310.05859375,  32054138.18359375,  32102966.30859375,
         32347106.93359375,  32395935.05859375,  32444763.18359375,
         32493591.30859375,  32737731.93359375,  32786560.05859375,
         32835388.18359375,  32884216.30859375,  33128356.93359375,
         33177185.05859375,  33226013.18359375,  33274841.30859375,
         33518981.93359375,  33567810.05859375,  33616638.18359375,
         33665466.30859375]])

In [25]: t2.getcol('CHAN_WIDTH')
Out[25]: 
array([[ 48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125]])

L*31MHz_TC0.time_split-split:

In [28]: t3.getcol('REF_FREQUENCY')
Out[28]: array([ 31834411.62109375])

In [29]: t3.getcol('TOTAL_BANDWIDTH')
Out[29]: array([ 1953125.])

In [30]: t3.getcol('CHAN_FREQ')
Out[30]: 
array([[ 30003356.93359375,  30052185.05859375,  30101013.18359375,
         30149841.30859375,  30393981.93359375,  30442810.05859375,
         30491638.18359375,  30540466.30859375,  30784606.93359375,
         30833435.05859375,  30882263.18359375,  30931091.30859375,
         31175231.93359375,  31224060.05859375,  31272888.18359375,
         31321716.30859375,  31565856.93359375,  31614685.05859375,
         31663513.18359375,  31712341.30859375,  31956481.93359375,
         32005310.05859375,  32054138.18359375,  32102966.30859375,
         32347106.93359375,  32395935.05859375,  32444763.18359375,
         32493591.30859375,  32737731.93359375,  32786560.05859375,
         32835388.18359375,  32884216.30859375,  33128356.93359375,
         33177185.05859375,  33226013.18359375,  33274841.30859375,
         33518981.93359375,  33567810.05859375,  33616638.18359375,
         33665466.30859375]])

In [31]: t3.getcol('CHAN_WIDTH')
Out[31]: 
array([[ 48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125,
         48828.125,  48828.125,  48828.125,  48828.125,  48828.125]])

@AHorneffer I'm doing exactly the same thing that the standard preprocessing pipeline does - I modeled my preprocessing parset on the history table of a more recent LBA observation (that has worked with PiLL). It uses the preflagger and no filtering as far as I can tell.

The data are from Cycle 0, so perhaps the channel/SB setup was different then from what it is now (120 SB vs 244 for example). The raw data for this observation (64 ch/SB, 1 sec integration) are available and I've preprocessed them as described. The channel/SB setup is at least consistent across this whole observation, even if it is different from current practice. Is there anything else 'special' about Cycle 0 data that might cause this issue?

History from LC6_007 target preprocessing pipeline (relevant snippet):

msin=/data/projects/LC6_007/L569147/uv/L569147_SAP000_SB000_uv.MS,
msin.autoweight=true,
msin.band=-1,
msin.baseline=,
msin.blrange=[],
msin.corrtype=,
msin.datacolumn=DATA,
msin.endtime=2017/02/18/12:59:59.995,
msin.forceautoweight=false,
msin.missingdata=false,
msin.nchan=nchan,
msin.orderms=false,
msin.sort=false,
msin.startchan=0,
msin.starttime=2017/02/18/11:00:00.000,
msin.useflag=true,

preflagger[0].abstime=[],
preflagger[0].azimuth=[],
preflagger[0].baseline=,
preflagger[0].blrange=[],
preflagger[0].chan=[0..nchan/32-1,31*nchan/32..nchan-1],
preflagger[0].corrtype=
preflagger[0].count.path=-,
preflagger[0].count.save=false,
preflagger[0].elevation=[],
preflagger[0].expr=,
preflagger[0].freqrange=[],
preflagger[0].lst=[],
preflagger[0].reltime=[],
preflagger[0].timeofday=[],
preflagger[0].timeslot=[],
preflagger[0].type=preflagger,

preflagger[1].abstime=[],
preflagger[1].azimuth=[],
preflagger[1].baseline=,
preflagger[1].blrange=[],
preflagger[1].chan=[],
preflagger[1].corrtype=auto,
preflagger[1].count.path=-,
preflagger[1].count.save=false,
preflagger[1].elevation=[],
preflagger[1].expr=,
preflagger[1].freqrange=[],
preflagger[1].lst=[],
preflagger[1].reltime=[],
preflagger[1].timeofday=[],
preflagger[1].timeslot=[],
preflagger[1].type=preflagger,

My preprocessing pipeline (relevant snippet):

msin=
msin.autoweight=true
msin.datacolumn=DATA
msin.forceautoweight=false
msin.missingdata=false
msin.orderms=false
msin.sort=false
msin.startchan=0
msin.useflag=false

preflagger0.chan=[0..nchan/32-1,31*nchan/32..nchan-1]
preflagger0.count.path=
preflagger0.count.save=true
preflagger0.type=preflagger

preflagger1.corrtype=auto
preflagger1.count.path=
preflagger1.count.save=true
preflagger1.type=preflagger
AHorneffer commented 7 years ago 
array([[ 30003356.93359375,  30052185.05859375,  30101013.18359375,
         30149841.30859375,  30393981.93359375,  30442810.05859375,
         30491638.18359375,  30540466.30859375,  30784606.93359375,
...

If you calculate the differences, then you get:

[ 48828.125, 48828.125, 48828.125, 244140.625, 48828.125, 48828.125, 48828.125, 244140.625]

So it looks like there are indeed full subbands missing. That is exactly what sort_times_into_freqGroups.py is supposed to fix.

[. . .]

Arrggh! I believe you observed only every other subband. So the smallest step between two subbands is two subbands. And the check if the bandwidth vs. step is too small doesn't trigger if the ratio is exactly 1/2, only if it is less than 1/2.

A question to @revoltek and @adrabent: if every second SB in a concatenated file is missing (and thus at least half the channels in the MS are flagged), is the data then usable for LBA processing? HBA data like this would not be usable for Factor, so the default of prefactor is to print out an error message and to refuse to continue processing unusable data. Would this also be an acceptable behaviour for PILL?

mknapp55 commented 7 years ago

@AHorneffer I guess that would explain why I have 120 SB spanning the observing bandwidth instead of the 244 for newer observations. Was this standard in Cycle 0? I was not involved with setting this observation up.

AHorneffer commented 7 years ago

@mknapp55: I don't know what was or is standard for LBA observations. In HBA we were more likely to observe in "bands": groups of consecutive subbands with gaps between the groups, like e.g. for MSSS.

If you want the script to allow for more than half the data being filled with dummy data, then one could hard-code the difference between two subbands (one thing that I wanted to avoid). In this case set:

freq_width = 195312.5

and kick out the related error-checking.

mknapp55 commented 7 years ago

@AHorneffer Is the hard-coding you suggest necessary given this old data I'm working with? If so, where do I put it and what is it replacing?

Currently, LBA observations put half of all available SB on the cal field and half on the target field. The cal SBs are usually 244-487 and the target SBs are 000-243. Despite the numbering, I believe they're interleaved - every (let's say) even SB is on the calibrator and every odd SB is on the target. This setup means there's a 1 SB gap in between all calibrator SBs and all target SBs. Perhaps early setups (Cycle 0) skipped more than one SB, or maybe the SBs were wider than they are now?

My point is that 50% (exactly) of the data in either the calibrator or target will have to be dummy data if you want frequency continuity across the band - this is under normal, standard observing setups. That hasn't been a problem for more recent (Cycle 4 onward) LBA observations processed with PiLL - they work just fine. It's only this older data that's causing problems.

Maybe none of this is relevant or I've misunderstood something - feel free to ignore if so.

mknapp55 commented 7 years ago

@adrabent Related: if there's a file missing from my set of target SBs (because it never got ingested into the LTA, for example), but the corresponding cal SB exists, can PiLL handle that? Or do I need to get rid of the corresponding cal SB before I try to run the pipeline? If PiLL isn't set up to handle that, it would be nice if it could give a warning and then fill in the missing SB with dummy flagged data and continue.

If it's necessary to get rid of the matching cal SB, can I just delete the appropriate L*flagcal and run from time_split, or do I need to delete the original MS from the input directory and rerun from scratch?

adrabent commented 7 years ago

@mknapp55 PiLL checks which calibrator solutions are available and will pick only those subbands from the target input directory. This means, missing calibrator solutions should lead to missing processed target files and should be filled with the dummy data at the concat step. At least this is the intended behaviour. @AHorneffer So far, PiLL does not separate the whole band into groups. It always uses the full bandwidth. So, basically, the data might still be usable if there is still enough signal-to-noise to calibrate it.

AHorneffer commented 7 years ago

@AHorneffer Is the hard-coding you suggest necessary given this old data I'm working with?

Necessary, but maybe not enough.

Despite the numbering, I believe they're interleaved - every (let's say) even SB is on the calibrator and every odd SB is on the target.

No, probably not. Please read up on the difference between beamlet number and frequency at which this beamlet is observed.

The _SBXXX_ number in the file-names (which is also called "Subband" on the LTA website) is really the beamlet number. In ILT observations the direction on the sky of each beamlet is given by the "SubArray Pointing" (as the LTA webpage calls it), the _SAPYYY_ number in the file-names. The ILT software allows for only 8 different subarray pointings, but the frequency of each beamlet can be freely chosen.

mknapp55 commented 7 years ago

@adrabent Makes sense in the case where there are missing cal files that are not missing in the target (SB244 does not exist, but SB000 does), but what about the reverse? For example, what if calibrator SB244 exists, but target SB000 does not? I've gotten errors when PiLL can't find the matching target SB for a calibrator SB that does exist and has been processed. In the past, I've been able to go get the missing file and fill in the gap, but I've recently run across a case where the missing file simply doesn't exist on the LTA. My question is how to handle this - can/should I delete the corresponding calibrator file (SB244 for example) and rerun the pipeline from time_split? Or do I have to delete the missing calibrator file and then run from the beginning of the calibrator step?

@AHorneffer I understand the difference between the beamlet number, observing frequency, and subarray pointing, though my terminology may have been sloppy. In all my LBA observations, SAP000 is the SubArray Pointing dedicated to the target and SAP001 is the SubArray Pointing dedicated to the calibrator. SAP000 has beamlets SB000-SB243 (or SB000-SB119 in the old data) and SAP001 has beamlets SB244-SB487 (SB120-SB239 in old data).

Obviously the old data that skips subbands (see example below) won't have as much SNR as the newer data that doesn't skip subbands, but I still want to process it. If that means filling in those missing spaces with dummy flagged data, that's fine. I just need the pipeline to handle that without erroring out.

Here's a quick example of the subband skipping in the old data (Cycle 0) vs newer data (Cycle 4). Old data (ch1 = SB001, ch2 = SB002):

In [51]: ch1 = ttarg_old.getcol('CHAN_FREQ')

In [52]: ch1
Out[52]:
array([[ 30381774.90234375,  30406188.96484375,  30430603.02734375,
         30455017.08984375,  30479431.15234375,  30503845.21484375,
         30528259.27734375,  30552673.33984375]])

In [53]: ch2 = ttarg_old2.getcol('CHAN_FREQ')

In [54]: ch2
Out[54]:
array([[ 30772399.90234375,  30796813.96484375,  30821228.02734375,
         30845642.08984375,  30870056.15234375,  30894470.21484375,
         30918884.27734375,  30943298.33984375]])

In [56]: ch2-ch1
Out[56]:
array([[ 390625.,  390625.,  390625.,  390625.,  390625.,  390625.,
         390625.,  390625.]])

In [58]: ch1[0][1]-ch1[0][0]
Out[58]: 24414.0625

In [59]: ch2[0][0]-ch1[0][7]
Out[59]: 219726.5625

New data (ch1_n = SB001, ch2_n = SB002):

In [62]: ch1_n = ttarg_new.getcol('CHAN_FREQ')

In [64]: ch1_n
Out[64]:
array([[ 30186462.40234375,  30210876.46484375,  30235290.52734375,
         30259704.58984375,  30284118.65234375,  30308532.71484375,
         30332946.77734375,  30357360.83984375]])

In [63]: ch2_n = ttarg_new2.getcol('CHAN_FREQ')

In [66]: ch2_n
Out[66]:
array([[ 30381774.90234375,  30406188.96484375,  30430603.02734375,
         30455017.08984375,  30479431.15234375,  30503845.21484375,
         30528259.27734375,  30552673.33984375]])

In [67]: ch2_n-ch1_n
Out[67]:
array([[ 195312.5,  195312.5,  195312.5,  195312.5,  195312.5,  195312.5,
         195312.5,  195312.5]])

In [69]: ch1_n[0][1]-ch1_n[0][0]
Out[69]: 24414.0625

In [70]: ch2_n[0][0]-ch1_n[0][7]
Out[70]: 24414.0625
adrabent commented 7 years ago

@mknapp55

For example, what if calibrator SB244 exists, but target SB000 does not? I've gotten errors when PiLL can't find the matching target SB for a calibrator SB that does exist and has been processed.

I have not tested such a case yet, but in principle PiLL should be able to handle that. If not, please open up another issue.

AHorneffer commented 7 years ago

Obviously the old data that skips subbands (see example below) won't have as much SNR as the newer data that doesn't skip subbands, but I still want to process it. If that means filling in those missing spaces with dummy flagged data, that's fine. I just need the pipeline to handle that without erroring out.

For that you need the hardcoded step-size I mentioned earlier. (See pull request.)

mknapp55 commented 7 years ago

@AHorneffer Your modified script worked, though I haven't gotten to the bbs step yet due to #25 to see if the bbs error is gone.

mknapp55 commented 7 years ago

I've gotten Cycle 0 data up to the point where #27 kicks in, so I'll close this issue. Thanks!