search

ratt-ru / Stimela-classic

Containerized radio interferometry scripting framework -- NB: Classic version is no longer in active development, use stimela 2! See README for details.
GNU General Public License v2.0
28 stars 16 forks source link

# FATAL: while parsing bind path: wrong bind syntax #776

Open viralp opened 1 year ago

viralp commented 1 year ago

Hi

I am getting the following singularity error in stimela. log-A168_D_Config_reduction_script.txt log-A168_D_Config_reduction_script-imaging_round_init.txt

2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: ---------------------------------
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Stimela version 1.7.6
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Running: A168 D Config reduction script
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: ---------------------------------
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Config directory cannot be found. Will create /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/stimela_parameter_files
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Adding cab '/export/home/viralp/singularity_images/stimela_wsclean_1.7.4.sif' (3.1) to recipe, container name 'imaging_round_init'
2023-04-05 15:58:41 STIMELA.imaging_round_init INFO: job started at 2023-04-05 15:58:41.343580
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: STEP 1 :: imaging_round_init:: Make initial image
2023-04-05 15:58:41 STIMELA INFO: Validating parameters for cab wsclean (stimela/wsclean:1.7.4)
2023-04-05 15:58:41 STIMELA INFO: Parameters validated and saved to /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/stimela_parameter_files/imaging_round_init-14042593832337616807319211861491.json
2023-04-05 15:58:41 STIMELA.imaging_round_init INFO: Starting container [imaging_round_init-14042593832337616807319211861491]. Timeout set to -1. The container ID is printed below.
# running cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall  --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/stimela_parameter_files/imaging_round_init-14042593832337616807319211861491.json:/stimela_mount/configfile:ro --bind /users/vparekh/venv3/lib/python3.8/site-packages/stimela/cargo/cab/wsclean/src:/stimela_mount/code:ro --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/passwd:/etc/passwd:rw --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/group:/etc/group:rw --bind /users/vparekh/venv3/bin/stimela_runscript:/singularity:ro --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/msdir:/stimela_mount/msdir:rw --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/input:/stimela_mount/input:ro --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/output_A168_relic_target:/stimela_mount/output:rw --bind /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/output_A168_relic_target/tmp:/stimela_mount/output/tmp:rw /export/home/viralp/singularity_images/stimela_wsclean_1.7.4.sif /singularity
# FATAL:   while parsing bind path: wrong bind syntax: /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345/passwd:/etc/passwd:rw
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall returns error code 255
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: job failed at 2023-04-05 15:58:41.426185 after 0:00:00.082605
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: Traceback (most recent call last):
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     job.run_job()
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2023-04-05 15:58:41 STIMELA.imaging_round_init ERROR: stimela.utils.StimelaCabRuntimeError: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall returns error code 255
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Completed jobs : []
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Remaining jobs : []
2023-04-05 15:58:41 STIMELA.A168 D Config reduction script INFO: Saving pipeline information in .last_A168_D_Config_reduction_script.json
Traceback (most recent call last):
  File "/users/vparekh/venv3/bin/stimela", line 11, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 409, in main
    _cmd(argv)
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 199, in run
    exec(stdr.read(), _globals)
  File "<string>", line 69, in <module>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run
    raise PipelineException(exc, self.completed, job, self.remaining) from None
stimela.exceptions.PipelineException: Job 'imaging_round_init:: Make initial image' failed: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16807319211808345 --containall returns error code 255

stimela_script_vla.txt

o-smirnov commented 1 year ago

This looks to be the real error, singularity doesn't understand the command line Stimela forms up:

while parsing bind path: wrong bind syntax:

@viralp what version of singularity is on this box? I presume it's either too old (not our problem then) or too new (in which case it is...)

viralp commented 1 year ago

Seems to be old version. 

singularity version 3.6.1+3-gb94fa3544
viralp commented 1 year ago

Ok singularity has been updated to 3.11. Now getting the following error

#  == Constructing PSF ==
# Detected 62.4 GB of system memory, usage not limited.
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/A168_VLA_L_D_targets.ms
# Determining min and max w & theoretical beam size... Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 69, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['wsclean', '-name', '/stimela_mount/output/17A-187.sb33657704.eb33666723.57871.671216585644init', '-j', '32', '-mem', '100', '-weight', 'briggs', '-0.5', '-size', '6000', '6000', '-scale', '0.06arcsec', '-channels-out', '32', '-no-small-inversion', '-pol', 'I', '-data-column', 'DATA', '-niter', '2000000', '-auto-threshold', '0.5', '-auto-mask', '5.0', '-mgain', '0.9', '-join-channels', '-fit-spectral-pol', '4', '-nmiter', '20', '/stimela_mount/msdir/A168_VLA_L_D_targets.ms']' died with <Signals.SIGKILL: 9>.
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: job failed at 2023-04-18 14:30:08.140287 after 0:00:32.468797
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: Traceback (most recent call last):
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     job.run_job()
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:   File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: stimela.utils.StimelaCabRuntimeError: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1
2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Completed jobs : []
2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Remaining jobs : []
2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Saving pipeline information in .last_A168_Config_reduction_script__Sequential_.json
Traceback (most recent call last):
  File "/users/vparekh/venv3/bin/stimela", line 11, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 409, in main
    _cmd(argv)
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 199, in run
    exec(stdr.read(), _globals)
  File "<string>", line 76, in <module>
  File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run
    raise PipelineException(exc, self.completed, job, self.remaining) from None
stimela.exceptions.PipelineException: Job 'imaging_round_init:: Make initial image' failed: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1
bennahugo commented 1 year ago

Don't think this has anything to do with singularity - it definitely runs and logs. Most likely the out of memory killer. Contact your system admin to adjust cgroups settings for your machine if there are limits set.

On Tue, 18 Apr 2023, 22:45 Viral Parekh, @.***> wrote:

Ok singularity has been updated to 3.11. Now getting the following error

== Constructing PSF ==

Detected 62.4 GB of system memory, usage not limited.

Opening reordered part 0 spw 0 for /stimela_mount/msdir/A168_VLA_L_D_targets.ms

Determining min and max w & theoretical beam size... Traceback (most recent call last):

File "/stimela_mount/code/run.py", line 69, in

subprocess.check_call(shlex.split(_runc))

File "/usr/lib/python3.8/subprocess.py", line 364, in check_call

raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['wsclean', '-name', '/stimela_mount/output/17A-187.sb33657704.eb33666723.57871.671216585644init', '-j', '32', '-mem', '100', '-weight', 'briggs', '-0.5', '-size', '6000', '6000', '-scale', '0.06arcsec', '-channels-out', '32', '-no-small-inversion', '-pol', 'I', '-data-column', 'DATA', '-niter', '2000000', '-auto-threshold', '0.5', '-auto-mask', '5.0', '-mgain', '0.9', '-join-channels', '-fit-spectral-pol', '4', '-nmiter', '20', '/stimela_mount/msdir/A168_VLA_L_D_targets.ms']' died with <Signals.SIGKILL: 9>.

2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: job failed at 2023-04-18 14:30:08.140287 after 0:00:32.468797 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: Traceback (most recent call last): 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: job.run_job() 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: self.job.run(output_wrangler=self.apply_output_wranglers) 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall", 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) 2023-04-18 14:30:08 STIMELA.imaging_round_init ERROR: stimela.utils.StimelaCabRuntimeError: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1 2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Completed jobs : [] 2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Remaining jobs : [] 2023-04-18 14:30:08 STIMELA.A168 Config reduction script (Sequential) INFO: Saving pipeline information in .last_A168_Config_reduction_script_Sequential.json Traceback (most recent call last): File "/users/vparekh/venv3/bin/stimela", line 11, in main.main([a for a in sys.argv[1:]]) File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 409, in main _cmd(argv) File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/main.py", line 199, in run exec(stdr.read(), _globals) File "", line 76, in File "/users/vparekh/venv3/lib/python3.8/site-packages/stimela/recipe.py", line 764, in run raise PipelineException(exc, self.completed, job, self.remaining) from None stimela.exceptions.PipelineException: Job 'imaging_round_init:: Make initial image' failed: cd /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 && singularity run --workdir /export/home/viralp/A168/17A-187.sb33657704.eb33666723.57871.671216585644_d/test2/stimela/.stimela_workdir-16818497755035388 --containall returns error code 1

— Reply to this email directly, view it on GitHub https://github.com/ratt-ru/Stimela/issues/776#issuecomment-1513772559, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4RE6RVO5J2M32GVXCTPGTXB34LLANCNFSM6AAAAAAWUUFOMM . You are receiving this because you are subscribed to this thread.Message ID: @.***>