std exception detected: Frequency 187211608.886719 not found in target/TGSSphase

[cyriltasse]

Hi

I got these error messages when running the Delay-Calibration.parset

std exception detected: Frequency 187211608.886719 not found in target/TGSSphase

2023-04-11 10:05:09 ERROR   node.nancep7.obs-nancay.fr.executable_args: Command '/opt/lofar/lofar/bin/NDPPP' returned non-zero exit status 1
2023-04-11 10:05:10 WARNING genericpipeline.executable_args: 
2023-04-11 10:05:10 ERROR   genericpipeline.executable_args: Remote process python /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/recipes/nodes/executable_args.py ['/data/levangelista/NGC4151/TARGET/L402151_SB487_uv.dppp.MS', '/opt/lofar/lofar/bin/NDPPP', [], {'applybandpass.parmdb': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration/solutions.h5', 'applybandpass.correction': 'bandpass', 'msout.writefullresflag': 'False', 'applybeam.type': 'applybeam', 'msout.datacolumn': 'DATA', 'applyphase.type': 'applycal', 'applyphase.parmdb': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration/solutions.h5', 'filter.baseline': '*&;!CS026HBA0*;!CS026HBA1*', 'applyRM.parmdb': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration/solutions.h5', 'applyPA.type': 'applycal', 'applybandpass.type': 'applycal', 'applyPA.parmdb': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration/solutions.h5', 'msin': '/data/levangelista/NGC4151/TARGET/L402151_SB487_uv.dppp.MS', 'applyclock.parmdb': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration/solutions.h5', 'msin.datacolumn': 'DATA', 'applyclock.correction': 'clock', 'msin.baseline': '*&;!CS026HBA0*;!CS026HBA1*', 'filter.remove': 'true', 'applybeam.updateweights': 'True', 'applyclock.type': 'applycal', 'flag.type': 'preflagger', 'applyPA.solset': 'calibrator', 'applyPA.correction': 'polalign', 'flag.baseline': '[ ]', 'applybandpass.updateweights': 'True', 'msout.storagemanager': 'dysco', 'applyRM.type': 'applycal', 'applyphase.solset': 'target', 'applyphase.correction': 'TGSSphase', 'applybandpass.solset': 'calibrator', 'flagamp.amplmin': '1e-30', 'numthreads': '2', 'applyRM.correction': 'RMextract', 'filter.type': 'filter', 'applybeam.usechannelfreq': 'True', 'steps': '[flag,flagamp,filter,applyPA,applybandpass,applyclock,applybeam,applyRM,applyphase,count]', 'msout': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration/L402151_SB487_uv.dppp.ndppp_prep_target', 'flagamp.type': 'preflagger', 'applyRM.solset': 'target', 'applyclock.solset': 'calibrator'}, '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Delay-Calibration', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'lofar', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'LOFARDATAROOT': '/opt/lofar/data', 'USER_PATH': '/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/cep/lofar/bin:/cep/lofar/scripts:/opt/dell/srvadmin/bin:/home/levangelista/.local/bin:/home/levangelista/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin', 'PYTHONPATH': '/opt/lofar:/opt/lofar/ddf-pipeline/scripts:/opt/lofar/ddf-pipeline/utils:/opt/lofar/DDFacet/lib/python2.7/site-packages:/opt/lofar/killMS:/opt/lofar/killMS/Predict:/opt/lofar/killMS/Array:/opt/lofar/killMS/Array/Dot:/opt/lofar/killMS/Gridder:/opt/lofar/DDFacet/bin:/opt/lofar/DynSpecMS:/opt/lofar/DPPP/lib/python3.7/site-packages:/opt/lofar/DPPP/usermodules:/opt/lofar/lofar/lib64/python2.7/site-packages', 'OMP_NUM_THREADS': '8', 'LOFARROOT': '/opt/lofar/lofar', 'PATH': '/opt/lofar/pyenv-py2/bin:/opt/lofar/DDFacet/bin:/opt/lofar/DDFacet/src/SkyModel:/opt/lofar/DynSpecMS/:/opt/lofar/killMS:/opt/lofar/ddf-pipeline/scripts:/usr/lib64/openmpi/bin:/opt/lofar/wsclean/bin:/opt/lofar/lofar/bin:/opt/hdf5/bin:/opt/lofar/dysco/bin:/opt/lofar/difmap/uvf_difmap:/opt/lofar/DPPP/bin:/opt/lofar/ds9/bin:/opt/lofar/casacore/bin:/opt/lofar/aoflagger/bin:/opt/montage/Montage-6.0/bin:/usr/local/cuda/bin:/opt/hdf5/bin:/opt/lofar/lofar/sbin:/opt/lofar/lofar/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LD_LIBRARY_PATH': '/opt/lofar/aoflagger/lib:/opt/lofar/casacore/lib:/opt/lofar/DPPP/lib:/opt/lofar/dysco/lib:/opt/lofar/EveryBeam/lib:/opt/hdf5/lib:/opt/lofar/idg/lib:/opt/lofar/lofar/lib:/opt/lofar/lofar/lib64:/opt/lofar/LOFARBeam/lib:/opt/lofar/lofarstman/lib64:/usr/local/cuda/lib64:/opt/hdf5/lib:/opt/intel/mkl/lib/intel64/:/usr/lib64/openmpi/lib/:/opt/lofar/lofar/lib64:/.singularity.d/libs'}] failed on localhost

etc...

I checked at the [...]/Pre-Facet-Target/results/cal_values/solutions.h5 and saw

In [11]: H.root.target.TGSSphase.val[:].shape
Out[11]: (3595, 67, 23, 2)

In [13]: H.root.target.TGSSphase.freq[:].shape
Out[13]: (23,)

In [12]: H.root.target.TGSSphase.freq[:]
Out[12]: 
array([1.21189880e+08, 1.23143005e+08, 1.25096130e+08, 1.27049255e+08,
       1.29002380e+08, 1.30955505e+08, 1.32908630e+08, 1.34861755e+08,
       1.36814880e+08, 1.38768005e+08, 1.40721130e+08, 1.42674255e+08,
       1.44627380e+08, 1.46580505e+08, 1.48533630e+08, 1.50486755e+08,
       1.52439880e+08, 1.54393005e+08, 1.56346130e+08, 1.58299255e+08,
       1.60252380e+08, 1.62205505e+08, 1.64158630e+08])

I checked and my calibrator and target MSs have exactly the same frequencies (240 MS between 120310974.12109375 and 187303161.62109375). This is how the frequencies of the dataset and TGSSphase.freq differ

I guess there is some filtering going on somewhere - maybe I need to remove some MSs and restart the processing?

[cyriltasse]

I checked in the various parsets and did not see where the fmin/fmax of the calibration were set...

[vmahat]

It's possible that SBs > 168 MHz don't have solutions, as more than 50% of it got flagged during prefactor. I would remove SBs at 168 MHz and higher and re-run, or reduce the `! min_unflagged_fraction = 0.5' in the prefactor parset and re-run if you really want those frequencies.

[cyriltasse]

Well done! :)) I've moved this files elsewhere and I'm now rerunning the Delay-Calibration.parset from scratch (not rerunning Pre-Facet-Calibrator.parset and Pre-Facet-Target.parset) - do you think it's fine?

[vmahat]

I presume this would be fine...

[tikk3r]

do you think it's fine?

Yep that's fine. Stuff above 168 MHz is usually discarded for this reason.

[lmorabit]

Hmmm, this is a bit strange, because the pipeline has a step that specifically checks the prefactor solutions for frequency coverage. Do you have the prefactor solutions somewhere?

[cyriltasse]

Hmmm, this is a bit strange, because the pipeline has a step that specifically checks the prefactor solutions for frequency coverage. Do you have the prefactor solutions somewhere?

The trick is good enough for me for now but I have posted the h5 here if you want to take a look. I close for now! Thanks for the support!