search

tikk3r / flocs

Containers recipes for software stacks used in LOFAR data reduction.
https://tikk3r.github.io/flocs/
GNU General Public License v3.0
5 stars 6 forks source link

Issue with run_LINC_target #84

Open nealjackson opened 1 week ago

nealjackson commented 1 week ago

I'm having a problem running the run_LINC_target runner from this repository, with the following invocation:

sbatch -J new1 run_LINC_target.sh -s /share/nas/njj/singularity/lofar_sksp_v4.4.0_cascadelake_cascadelake_ddf_mkl_cuda.sif -d /share/nas/njj/lofar-vlbi-linctarget/701305 -b /home/njj/lofar,/share/nas/njj/lofar-vlbi-linctarget,/share/nas/njj/LINC,/share/nas/njj/flocs/flocs/runners -c /share/nas/njj/lofar-vlbi-linctarget/701305/LINC-cal_solutions.h5

If I do it as is, it gives me two errors:

usage: create_ms_list.py [-h] {LINC,VLBI} ... create_ms_list.py: error: argument parser: invalid choice: '/share/nas/njj/lofar-vlbi-linctarget/701305' (choose from 'LINC', 'VLBI') LINC starting INFO /opt/lofar/pyenv-py3/bin/cwltool 3.1.20230425144158 INFO Resolved '/share/nas/njj/LINC/workflows/HBA_target.cwl' to 'file:///share/nas/njj/LINC/workflows/HBA_target.cwl' ERROR Tool definition failed validation: [Errno 2] No such file or directory: '/share/nas/njj/lofar-vlbi-linctarget/tmp.dpUZAZgSoc/mslist.json' If I change the script to put 'LINC target' after the create_ms_list.py it reads the datafiles OK, but I then still get an error finding the mslist.json:

INFO Resolved '/share/nas/njj/LINC/workflows/HBA_target.cwl' to 'file:///share/nas/njj/LINC/workflows/HBA_target.cwl'
ERROR Tool definition failed validation:
[Errno 2] No such file or directory: '/share/nas/njj/lofar-vlbi-linctarget/tmp.wvbXrvL2fd/mslist.json'

There is however the following file:

/share/nas/njj/lofar-vlbi-linctarget/L701305_LINC_target/tmp.wvbXrvL2fd/mslist_LINC_target.json

Is there a simple fix? I think I might be using this incorrectly....

tikk3r commented 1 week ago

Hi Neal,

create_ms_list.py has seen a bunch of changes over the past months indeed. As you've noticed the syntax has changed to require a pipeline and workflow as second and third arguments now.

Runner wise, could it be that you are using an old version? Looking at its current form https://github.com/tikk3r/flocs/blob/fedora-py3/runners/run_LINC_target_HBA.sh it should correctly grab mslist_LINC_target.json instead of looking for the old mslist.json.