search

lofar-astron / factor

Facet calibration for LOFAR
http://www.astron.nl/citt/facet-doc
GNU General Public License v2.0
19 stars 12 forks source link

<class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'> #172

Closed FedericaSavini closed 7 years ago

FedericaSavini commented 7 years ago

Hello everybody,

I am running prefactor on a group of 10 sb for my target and I get an error I've never seen before: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'>

pipeline-2.txt

Backtrace follows:

0 0x7f25cc0c372e in LOFAR::DPPP::ApplyCal::updateInfo(LOFAR::DPPP::DPInfo const&) at ApplyCal.cc:130

1 0x7f25cc083595 in LOFAR::DPPP::DPStep::setInfo(LOFAR::DPPP::DPInfo const&) at DPStep.cc:38

2 0x7f25cc0835e2 in LOFAR::DPPP::DPStep::setInfo(LOFAR::DPPP::DPInfo const&) at DPStep.cc:39

3 0x7f25cc04bafc in LOFAR::DPPP::DPRun::makeSteps(LOFAR::ParameterSet const&) at DPRun.cc:339

4 0x7f25cc0481df in LOFAR::DPPP::DPRun::execute(std::string const&, int, char**) at DPRun.cc:132

5 0x407c70 in main at NDPPP.cc:88

2016-12-16 16:14:27 ERROR node.node05.executable_args: Command '/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/bin/NDPPP' returned non-zero exit status 1 2016-12-16 16:14:27 INFO node.node05.executable_args: Total time 21.0746s; user time: 0.0895s; system time: 0.2091s 2016-12-16 16:14:27 DEBUG node.node05.executable_args: Start time was 1481901246.7999s; end time was 1481901267.8746s 2016-12-16 16:14:27 DEBUG genericpipeline.executable_args: Results for job 0 submitted by ('192.168.79.5', 57370) 2016-12-16 16:14:28 DEBUG genericpipeline.executable_args: 2016-12-16 16:14:28 WARNING genericpipeline.executable_args: 2016-12-16 16:14:28 INFO genericpipeline.executable_args: Subprocess completed with exit status 1: /bin/sh -c python /home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/lib/python2.7/site-packages/lofarpipe/recipes/nodes/executable_args.py 0 192.168.79.5 47133 2016-12-16 16:14:28 ERROR genericpipeline.executable_args: Remote process python /home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/lib/python2.7/site-packages/lofarpipe/recipes/nodes/executable_args.py ['/beegfs/stnv039/A2261/Pre-Facet-Cal-Target/group01/Pre-Facet-Cal-Target/L527549_SB244_uv.dppp_1289BD101t_121MHz.msdpppconcat', '/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/bin/NDPPP', [], {'msin.datacolumn': 'DATA', 'applycal.type': 'applycal', 'numthreads': '8', 'msout.writefullresflag': 'False', 'applycal.parmdb': '/beegfs/stnv039/A2261/Pre-Facet-Cal-Target/group01/Pre-Facet-Cal-Target/L527549_SB244_uv.dppp_1289BD101t_121MHz.msdpppconcat/instrument_directionindependent', 'steps': '[applycal]', 'msin': '/beegfs/stnv039/A2261/Pre-Facet-Cal-Target/group01/Pre-Facet-Cal-Target/L527549_SB244_uv.dppp_1289BD101t_121MHz.msdpppconcat', 'msout.datacolumn': 'CORRECTED_DATA', 'applycal.correction': 'gain', 'msout': '/beegfs/stnv039/A2261/Pre-Facet-Cal-Target/group01/Pre-Facet-Cal-Target/L527549_SB244_uv.dppp_1289BD101t_121MHz.msdpppconcat'}, '/beegfs/stnv039/A2261/Pre-Facet-Cal-Target/group01/Pre-Facet-Cal-Target', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'lofar', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'LOFARDATAROOT': '/opt/lofar/data', 'OMP_NUM_THREADS': '8', 'PYTHONPATH': '/home/lofar/local2/lib/python:/home/lofar/local/src/factor_master:/home/lofar/local/lib/python:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone/aux-lib:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone:/home/lofar/local/src/DistCal:/home/lofar/local/src/LSMTool:/home/lofar/local/src/LoSoTo:/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/lib/python2.7/site-packages:/home/lofar/lofar_build/lofar/release/lib/python2.7/site-packages:/home/lofar/lofar_build/local/release/lib/python2.7:/home/lofar/lofar_build/lofar/release/lib/python2.7/dist-packages:/home/lofar/local2/lib/python:/home/lofar/local/src/factor:/home/lofar/local/lib/python:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone/aux-lib:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone:/home/lofar/local/src/DistCal:/home/lofar/local/src/LSMTool:/home/lofar/local/src/LoSoTo:/home/lofar/lofar_build/lofar/release/lib/python2.7/site-packages:/home/lofar/lofar_build/local/release/lib/python2.7:/home/lofar/lofar_build/lofar/release/lib/python2.7/dist-packages:', 'LOFARROOT': '/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed', 'PBS_O_PATH': '/home/lofar/local/bin:/home/lofar/local/src/wsclean/chgcentre-1.1/build:/home/lofar/local/src/wsclean/wsclean-1.12f/wsclean/build:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone:/home/lofar/local/src/ds9:/home/lofar/local/src/aladin/Aladin:/home/lofar/local/src/topcat:/home/lofar/local/src/sagecal-code/src/MS:/opt/openmpi/1.8.1/gcc/bin:/home/lofar/local/src/factor/bin:/home/lofar/local/src/LSMTool/bin:/home/lofar/local/src/LoSoTo/bin:/home/lofar/local/casapy4.5:/home/lofar/lofar_build/local/release/bin:/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/sbin:/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/bin:/home/lofar/lofar_build/lofar/release/bin:/home/lofar/lofar_build/lofar/release/sbin:/home/lofar/lofar_build/local/release/bin:/home/lofar/lofar_build/local/release/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/torque/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/torque/bin', 'MANPATH': ':/opt/torque/share/man', 'PATH': '/home/lofar/local/bin:/home/lofar/local/src/wsclean/chgcentre-1.1/build:/home/lofar/local/src/wsclean/wsclean-1.12f/wsclean/build:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone:/home/lofar/local/src/ds9:/home/lofar/local/src/aladin/Aladin:/home/lofar/local/src/topcat:/home/lofar/local/src/sagecal-code/src/MS:/opt/openmpi/1.8.1/gcc/bin:/home/lofar/local/src/factor_master/bin:/home/lofar/local/src/LSMTool/bin:/home/lofar/local/src/LoSoTo/bin:/home/lofar/local/casapy4.5:/home/lofar/lofar_build/local/release/bin:/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/sbin:/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/bin:/home/lofar/lofar_build/lofar/release/bin:/home/lofar/lofar_build/lofar/release/sbin:/home/lofar/lofar_build/local/release/bin:/home/lofar/lofar_build/local/release/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/torque/bin:/home/lofar/local/bin:/home/lofar/local/src/wsclean/chgcentre-1.1/build:/home/lofar/local/src/wsclean/wsclean-1.12f/wsclean/build:/home/lofar/local/src/Survey_pipeline/lofarcodes-standalone:/home/lofar/local/src/ds9:/home/lofar/local/src/aladin/Aladin:/home/lofar/local/src/topcat:/home/lofar/local/src/sagecal-code/src/MS:/opt/openmpi/1.8.1/gcc/bin:/home/lofar/local/src/factor/bin:/home/lofar/local/src/LSMTool/bin:/home/lofar/local/src/LoSoTo/bin:/home/lofar/local/casapy4.5:/home/lofar/lofar_build/local/release/bin:/home/lofar/lofar_build/lofar/release/bin:/home/lofar/lofar_build/lofar/release/sbin:/home/lofar/lofar_build/local/release/bin:/home/lofar/lofar_build/local/release/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/torque/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/opt/torque/bin', 'LD_LIBRARY_PATH': '/home/lofar/local/src/dysco/build:/sw/compiler/gcc-4.9.2/lib64:/home/lofar/local/casacore-2.0.3/lib:/home/lofar/local/lib:/home/lofar/local/cula/lib64:/home/lofar/local/src/LOFAR_2.17/build/gnu_debug/installed/lib:/home/lofar/lofar_build/lofar/release/lib:/home/lofar/lofar_build/lofar/release/lib64:/home/lofar/lofar_build/local/release/lib:/home/lofar/lofar_build/local/release/lib64:/home/lofar/lofar_build/local/release/lib:/home/lofar/lofar_build/local/release/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib:/home/lofar/local/src/dysco/build:/sw/compiler/gcc-4.9.2/lib64:/home/lofar/local/casacore-2.0.3/lib:/home/lofar/local/lib:/home/lofar/local/cula/lib64:/home/lofar/lofar_build/lofar/release/lib:/home/lofar/lofar_build/lofar/release/lib64:/home/lofar/lofar_build/local/release/lib:/home/lofar/lofar_build/local/release/lib64:/home/lofar/lofar_build/local/release/lib:/home/lofar/lofar_build/local/release/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local/cuda/lib:'}] failed on localhost 2016-12-16 16:14:28 DEBUG genericpipeline.executable_args: compute.dispatch results job 0: job_duration: 22.5860099792, returncode: 1 2016-12-16 16:14:29 DEBUG genericpipeline.executable_args: Adding node_logging_information 2016-12-16 16:14:29 ERROR genericpipeline.executable_args: A job has failed with returncode 1 and error_tolerance is not set. Bailing out! 2016-12-16 16:14:29 WARNING genericpipeline.executable_args: Note: recipe outputs are not complete 2016-12-16 16:14:29 WARNING genericpipeline.executable_args: recipe executable_args completed with errors 2016-12-16 16:14:29 WARNING genericpipeline: dppp reports failure (using executable_args recipe) 2016-12-16 16:14:29 ERROR genericpipeline: 2016-12-16 16:14:29 ERROR genericpipeline: Failed pipeline run: Pre-Facet-Cal-Target 2016-12-16 16:14:29 ERROR genericpipeline: Detailed exception information: 2016-12-16 16:14:29 ERROR genericpipeline: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'> 2016-12-16 16:14:29 ERROR genericpipeline: dppp failed 2016-12-16 16:14:29 ERROR genericpipeline: 2016-12-16 16:14:29 ERROR genericpipeline: LOFAR Pipeline finished unsuccesfully. 2016-12-16 16:14:29 WARNING genericpipeline: recipe genericpipeline completed with errors

tammojan commented 7 years ago

The error just above says:

LOFAR Exception detected: [Exception: No gains found in parmdb /beegfs/stnv039/A2261/Pre-Facet-Cal-Target/group01/Pre-Facet-Cal-Target/L527549_SB244_uv.dppp_1289BD101t_121MHz.msdpppconcat/instrument_directionindependent] in function virtual...
FedericaSavini commented 7 years ago

Total NDPPP time 6037.9 real 26770.4 user 12.9 system 0.5% MSReader 0.0% Filter filter. 98.8% GainCal gaincal. 80.6% of it spent in predict 5.5% of it spent in reordering visibility data 13.8% of it spent in estimating gains and computing residuals 0.0% of it spent in writing gain solutions to disk Converged: 3595, stalled: 0, non converged: 0, failed: 0 Iters converged: 3, stalled: 0, non converged: 0, failed: 0 0.4% MSUpdater msout.

It looks like determining the solutions was ok, but they are not in the instrument table. On the second group the pipeline finished successfully so it's not my model...

darafferty commented 7 years ago

Closing due to lack of activity. Reopen if still an issue.