run without containers

[HannoSpreeuw]

make run on a cluster without docker gives Docker is not available for this tool, try --no-container to disable Docker So I tried make run --no-container but that gives make: unrecognized option '--no-container' So I tried make --no-container but that also gives make: unrecognized option '--no-container'

[mr-c]

Hello @HannoSpreeuw

The makefile is provided as a convenience. This error message is from either cwltool or toil-cwl-runner so you could try the suggested option appended to the real command run

For example, try make run -n to see the commands that would be run.

In this instance I suggest make run-udocker as an alternative to Docker or Singularity. Otherwise you would need to install each of the software requirements of the workflow yourself.

[HannoSpreeuw]

Thanks. I am afraid udocker is not available on this cluster: make run-udocker ..... ..... Error: create container: getting layers or json Error: in download: HTTP/1.1 400 Bad Request Error: create container: getting layers or json Error: image or container not available [job ndppp_prep_cal] Job error: Error collecting output for parameter 'msout': steps/ndppp_prep_cal.cwl:164:7: Did not find output file with glob pattern: '[u'calibrated.MS']' [job ndppp_prep_cal] completed permanentFail Workflow cannot make any more progress. Final process status is permanentFail make: *** [run-udocker] Error 1

[HannoSpreeuw]

But my colleague found this package for installation of udocker.

[HannoSpreeuw]

But is running prefactor-cwl without containers not possible at all? Oh sorry, I now understand what you mean with

Otherwise you would need to install each of the software requirements of the workflow yourself

Running without containers would involve a lot of extra work.

[mr-c]

Currently the Makefile installs the main udocker branch: https://github.com/EOSC-LOFAR/prefactor-cwl/blob/master/Makefile#L24

Can you try out the latest devel udocker version?

rm .virtualenv/bin/udocker
curl https://raw.githubusercontent.com/indigo-dc/udocker/devel/udocker.py > .virtualenv/bin/udocker
chmod u+rx .virtualenv/bin/udocker
.virtualenv/bin/udocker install

[mr-c]

Also, what is the version of curl that is installed?

[HannoSpreeuw]

[hspreeuw@fs5 prefactor-cwl]$ curl --version curl 7.49.0 (x86_64-pc-linux-gnu) libcurl/7.49.0 OpenSSL/1.0.2p zlib/1.2.11 Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

[HannoSpreeuw]

make run-udocker [workflow ] start [workflow ] starting step ndppp_prep_cal [step ndppp_prep_cal] start [job ndppp_prep_cal] Output of job will be cached in /home/hspreeuw/prefactor-cwl/cache/187eef3d15e77102e21a6feb47bb8d9f [job ndppp_prep_cal] /home/hspreeuw/prefactor-cwl/cache/187eef3d15e77102e21a6feb47bb8d9f$ /home/hspreeuw/prefactor-cwl/.virtualenv/bin/udocker \ --quiet \ run \ --volume=/home/hspreeuw/prefactor-cwl/cache/187eef3d15e77102e21a6feb47bb8d9f:/var/spool/cwl \ --volume=/tmp/tmplmvobV:/tmp \ --volume=/home/hspreeuw/prefactor-cwl/data/L570745_SB000_uv_first10.MS:/var/lib/cwl/stg44fa7578-bb33-475e-935a-45fc6824e3e3/L570745_SB000_uv_first10.MS \ --workdir=/var/spool/cwl \ --rm \ --env=TMPDIR=/tmp \ --env=HOME=/var/spool/cwl \ kernsuite/prefactor \ NDPPP \ msout=calibrated.MS \ average.freqresolution=48.82kHz \ avg.freqstep=2 \ average.timeresolution=4 \ avg.timestep=2 \ avg.type=average \ baseline=[CS013HBA] \ 'filter.baseline=CS, RS&&' \ filter.remove=True \ filter.type=filter \ 'flag.baseline=[ CS013HBA ]' \ flag.type=filter \ flagamp.amplmin=1e-30 \ flagamp.type=preflagger \ msin=/var/lib/cwl/stg44fa7578-bb33-475e-935a-45fc6824e3e3/L570745_SB000_uv_first10.MS \ msin.datacolumn=DATA \ msout.overwrite=True \ msout.writefullresflag=False \ steps=[flag,filter,avg,flagamp] Error: create container: getting layers or json NDPPP: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory [job ndppp_prep_cal] Job error: Error collecting output for parameter 'msout': steps/ndppp_prep_cal.cwl:164:7: Did not find output file with glob pattern: '[u'calibrated.MS']' [job ndppp_prep_cal] completed permanentFail Workflow cannot make any more progress. Final process status is permanentFail make: *** [run-udocker] Error 1

[HannoSpreeuw]

Needs lapack. Should be no problem.

[HannoSpreeuw]

module load lapack make run-udocker same error

[HannoSpreeuw]

Very weird. We have /usr/lib64/liblapack.so.3 export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH Still the same error.

[gijzelaerr]

Hi @HannoSpreeuw all these things have been more of a tech demo than a plug-and-play and ready-to-use solution. I'm not sure why udocker doesn't work, it looks like there might be a bug with it and it doesn't initialise the shared libraries properly inside the containers. Can't you use Singularity?

[mr-c]

Is this the default dataset, or did you edit anything?

Error: create container: getting layers or json NDPPP: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory

Means that liblapack inside the container was not found (for whatever reason) so any changes outside the container (like module load ... ) won't fix this issue.

[HannoSpreeuw]

That makes sense....duh. No I didn't edit anything. Just make small make run-udocker

[HannoSpreeuw]

Somehow this error disappeared when I got more disk space on the cluster. But that would not have any effect on lapack issues, one would think. However, there is an error message before the lapack error: Error: create container: getting layers or json

Now make run-udocker gives #23