search

lmorabit / lofar-vlbi

GNU General Public License v3.0
16 stars 13 forks source link

ERROR main() got an unexpected keyword argument 'helperscriptspath-h5merge' #94

Closed cyriltasse closed 1 year ago

cyriltasse commented 1 year ago

It now gets further - but it crashes with this error, when it calls for an h5merge script

FILE:  ['/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_121MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_123MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_125MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_127MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_129MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_130MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_132MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_134MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_136MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_138MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_140MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_142MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_144MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_146MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_148MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_150MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_152MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_154MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_156MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_158MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_160MHz.msdpppconcat', '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_162MHz.msdpppconcat']
Reading configuration file: pipeline.cfg
Reading task definition file(s): /opt/lofar/lofar/share/pipeline/tasks.cfg
2023-04-21 10:20:31 WARNING genericpipeline.executable_args: /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
2023-04-21 10:20:31 WARNING genericpipeline.executable_args: Debug: registered context Global=0
2023-04-21 10:20:36 WARNING genericpipeline.executable_args: 
Reading configuration file: pipeline.cfg
Reading task definition file(s): /opt/lofar/lofar/share/pipeline/tasks.cfg
2023-04-21 10:20:37 WARNING genericpipeline.executable_args: /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
2023-04-21 10:20:37 WARNING genericpipeline.executable_args: Debug: registered context Global=0
2023-04-21 10:23:04 WARNING genericpipeline.executable_args: 
Reading configuration file: pipeline.cfg
Reading task definition file(s): /opt/lofar/lofar/share/pipeline/tasks.cfg
2023-04-21 10:23:05 WARNING genericpipeline.executable_args: /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/support/utilities.pyc : Using default subprocess module!
2023-04-21 10:23:05 WARNING genericpipeline.executable_args: Debug: registered context Global=0
2023-04-21 10:23:05 ERROR   node.nancep7.obs-nancay.fr.python_plugin: main() got an unexpected keyword argument 'helperscriptspath-h5merge'
2023-04-21 10:23:05 WARNING genericpipeline.executable_args: 
2023-04-21 10:23:05 ERROR   genericpipeline.executable_args: Remote process python /opt/lofar/lofar/lib64/python2.7/site-packages/lofarpipe/recipes/nodes/python_plugin.py ['/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_121MHz_1273342CCt_141MHz.msdpppconcat', '/data/levangelista/DEV_VLBI/lofar-vlbi/bin/run_selfcal.py', ['/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/ILTJ120752.09+385559.1_L402151_SB244_uv.dppp_1273342BDt_121MHz_1273342CCt_141MHz.msdpppconcat'], {'helperscriptspath': '/data/levangelista/DEV_VLBI/lofar_facet_selfcal', 'configfile': '/data/levangelista/DEV_VLBI/lofar-vlbi/targetselfcal_config.txt', 'delay_sols': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions/../Delay-Calibration/delay_solve/merged_addCS_selfcalcyle009_linearfulljones_ILTJ120449.84+392947.8_L402151_SB244_uv.dppp_1273342BDt_121MHz_1273342CCt_141MHz.msdpppconcat.copy.phaseup.h5', 'helperscriptspath-h5merge': '/data/levangelista/DEV_VLBI/lofar_helpers', 'destdir': '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions'}, '/data/levangelista/LOFAR_VLBI_PIPELINE/RUN_DIR/Split-Directions', False, {'args_format_option_argument': '=', 'args_format_option': '-', 'args_format': 'gnu', 'args_formatlongoption': '--', 'args_format_argument': ''}, {'LOFARDATAROOT': '/opt/lofar/data', 'USER_PATH': '/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/cep/lofar/bin:/cep/lofar/scripts:/opt/dell/srvadmin/bin:/home/levangelista/.local/bin:/home/levangelista/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin', 'PYTHONPATH': '/opt/lofar:/opt/lofar/ddf-pipeline/scripts:/opt/lofar/ddf-pipeline/utils:/opt/lofar/DDFacet/lib/python2.7/site-packages:/opt/lofar/killMS:/opt/lofar/killMS/Predict:/opt/lofar/killMS/Array:/opt/lofar/killMS/Array/Dot:/opt/lofar/killMS/Gridder:/opt/lofar/DDFacet/bin:/opt/lofar/DynSpecMS:/opt/lofar/DPPP/lib/python3.7/site-packages:/opt/lofar/DPPP/usermodules:/opt/lofar/lofar/lib64/python2.7/site-packages', 'OMP_NUM_THREADS': '8', 'LOFARROOT': '/opt/lofar/lofar', 'PATH': '/opt/lofar/pyenv-py2/bin:/opt/lofar/DDFacet/bin:/opt/lofar/DDFacet/src/SkyModel:/opt/lofar/DynSpecMS/:/opt/lofar/killMS:/opt/lofar/ddf-pipeline/scripts:/usr/lib64/openmpi/bin:/opt/lofar/wsclean/bin:/opt/lofar/lofar/bin:/opt/hdf5/bin:/opt/lofar/dysco/bin:/opt/lofar/difmap/uvf_difmap:/opt/lofar/DPPP/bin:/opt/lofar/ds9/bin:/opt/lofar/casacore/bin:/opt/lofar/aoflagger/bin:/opt/montage/Montage-6.0/bin:/usr/local/cuda/bin:/opt/hdf5/bin:/opt/lofar/lofar/sbin:/opt/lofar/lofar/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LD_LIBRARY_PATH': '/opt/lofar/aoflagger/lib:/opt/lofar/casacore/lib:/opt/lofar/DPPP/lib:/opt/lofar/dysco/lib:/opt/lofar/EveryBeam/lib:/opt/hdf5/lib:/opt/lofar/idg/lib:/opt/lofar/lofar/lib:/opt/lofar/lofar/lib64:/opt/lofar/LOFARBeam/lib:/opt/lofar/lofarstman/lib64:/usr/local/cuda/lib64:/opt/hdf5/lib:/opt/intel/mkl/lib/intel64/:/usr/lib64/openmpi/lib/:/opt/lofar/lofar/lib64:/.singularity.d/libs'}] failed on localhost
2023-04-21 10:23:06 ERROR   genericpipeline.executable_args: All jobs failed. Bailing out!
2023-04-21 10:23:06 WARNING genericpipeline.executable_args: Note: recipe outputs are not complete
2023-04-21 10:23:06 WARNING genericpipeline.executable_args: recipe executable_args completed with errors
2023-04-21 10:23:06 WARNING genericpipeline: pythonplugin reports failure (using executable_args recipe)
2023-04-21 10:23:06 ERROR   genericpipeline: *******************************************
2023-04-21 10:23:06 ERROR   genericpipeline: Failed pipeline run: Split-Directions
2023-04-21 10:23:06 ERROR   genericpipeline: Detailed exception information:
2023-04-21 10:23:06 ERROR   genericpipeline: <class 'lofarpipe.support.lofarexceptions.PipelineRecipeFailed'>
2023-04-21 10:23:06 ERROR   genericpipeline: pythonplugin failed
2023-04-21 10:23:06 ERROR   genericpipeline: *******************************************

Looks like the error I got earlier - is there any simple fix to this?

tikk3r commented 1 year ago

Looks like a simple typo was made here: https://github.com/lmorabit/lofar-vlbi/blob/83945da648fdab2fcc1fc72b43c556032bcc5f9d/Split-Directions.parset#L202

where it should be helperscriptspath_h5merge instead of helperscriptspath-h5merge.

cyriltasse commented 1 year ago

Thanks! I fixed it one, and it crashes further on, I commented on yesterday's discussion with @lmorabit @jwpetley