full commands missing from log files

paoloserra commented 4 years ago

I no longer see the full command of the task in the corresponding log file.

E.g., log-self_cal-image_field0_iter0-20200410-155056.txt (which runs WSclean) reads as follows, and I do not see the full WSclean command in there.

Unless I'm doing something wrong, can we please get the full commands back. They're very. useful for debugging.

job started at 2020-04-10 15:51:21.522317
Instantiating container [image_field0_iter0-13968526240990415865338786744058]. The container ID is printed below.
running docker create --user 1001:1100  -v /24TB/pserra/caracal_venv/bin/stimela_docker_run:/docker_run:ro -v /24TB/pserra/caracal_venv/lib/python3.6/site-packages/stimela:/scratch/stimela:ro -v /24TB/pserra/vela2020/gps/stimela_parameter_files:/configs:ro -v /24TB/pserra/vela2020/gps/msdir:/home/pserra/msdir:rw -v /24TB/pserra/vela2020/gps/input:/input:ro -v /24TB/pserra/vela2020/gps/output:/home/pserra/output:rw -v /24TB/pserra/vela2020/gps/output/tmp:/home/pserra/output/tmp:rw  -e CONFIG=/configs/image_field0_iter0-13968526240990415865338786744058.json -e MSDIR=/home/pserra/msdir -e INPUT=/input -e HOME=/home/pserra/output -e OUTPUT=/home/pserra/output -e TMPDIR=/home/pserra/output/tmp --rm  --name image_field0_iter0-13968526240990415865338786744058 --shm-size 1gb pserra_cab/wsclean /bin/sh -c /docker_run
poll(): ready to read: [9]
c7535f9a5beeeae72be577c280378e8a6f1b0b40ac283b5803bd62f5f5f3f8c4
poll(): ready to read: [9, 11]
The docker create --user 1001:1100  -v /24TB/pserra/caracal_venv/bin/stimela_docker_run:/docker_run:ro -v /24TB/pserra/caracal_venv/lib/python3.6/site-packages/stimela:/scratch/stimela:ro -v /24TB/pserra/vela2020/gps/stimela_parameter_files:/configs:ro -v /24TB/pserra/vela2020/gps/msdir:/home/pserra/msdir:rw -v /24TB/pserra/vela2020/gps/input:/input:ro -v /24TB/pserra/vela2020/gps/output:/home/pserra/output:rw -v /24TB/pserra/vela2020/gps/output/tmp:/home/pserra/output/tmp:rw  -e CONFIG=/configs/image_field0_iter0-13968526240990415865338786744058.json -e MSDIR=/home/pserra/msdir -e INPUT=/input -e HOME=/home/pserra/output -e OUTPUT=/home/pserra/output -e TMPDIR=/home/pserra/output/tmp --rm  --name image_field0_iter0-13968526240990415865338786744058 --shm-size 1gb pserra_cab/wsclean /bin/sh -c /docker_run process has exited
Starting container [image_field0_iter0-13968526240990415865338786744058]. Timeout set to -1. The container ID is printed below.
running docker start -a image_field0_iter0-13968526240990415865338786744058
poll(): ready to read: [9]

poll(): ready to read: [9]
WSClean version 2.8 (2019-09-16)
poll(): ready to read: [9]
This software package is released under the GPL version 3.
poll(): ready to read: [9]
Author: André Offringa (offringa@gmail.com).
poll(): ready to read: [9]

poll(): ready to read: [9]
=== IMAGING TABLE ===
poll(): ready to read: [9]
       # Pol Ch JG ²G In Freq(MHz)
...

o-smirnov commented 4 years ago

@SpheMakh this is a stimela issue, right? It is just a question of echoing the command to the console in the run script, right?

I'm also puzzled by those poll() messages. I know where they come from (xrun_poll in Stimela), but they're issued at DEBUG level and so shouldn't be showing up by default, I think.

o-smirnov commented 4 years ago

See https://github.com/ratt-ru/Stimela/issues/552.

gigjozsa commented 4 years ago

@paoloserra is this still an issue?

o-smirnov commented 4 years ago

Fixed.

Cheers, Oleg

Sent from my phone. Quality of spelling inversely proportional to finger size.

On Sat, 25 Apr 2020, 22:23 Gyula Jozsa (Josh), notifications@github.com wrote:

@paoloserra https://github.com/paoloserra is this still an issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ska-sa/caracal/issues/944#issuecomment-619435692, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRLTP3G4PORZ3SPRXLRXGDRONBERANCNFSM4MFRNNUQ .

paoloserra commented 4 years ago

Full commands still missing from the log for some tasks. I've noticed AOFlagger and WSclean. The former was confirmed by @edeblok in https://github.com/caracal-pipeline/caracal/pull/1098#issuecomment-628420270 .

o-smirnov commented 4 years ago

There's a Stimela PR open that will address this, just pester @SpheMakh to get it in: https://github.com/ratt-ru/Stimela/pull/622.

paoloserra commented 3 years ago

The WSClean command is back in the log. AOflagger's is not.

paoloserra commented 2 years ago

Just checked this again. The AOFlagger command is not in the Caracal log (on singularity). Example:

2021-12-07 22:00:28 CARACal.Stimela.flag__cal INFO: STEP 5 :: flag__cal-rfi-ms0:: AOFlagger auto-flagging flagging pass ms=1638396089_sdp_l0-cal.ms fields=J0440-4333,J0408-6545
2021-12-07 22:00:28 CARACal.Stimela INFO: Validating parameters for cab autoflagger (stimela/aoflagger:1.2.0)
2021-12-07 22:00:28 CARACal.Stimela INFO: Parameters validated and saved to /scratch3/projects/fornax/data_reduction/mfs03/.stimela_workdir-16389071327985396/stimela_parameter_files/flag__cal_rfi_ms0-1401256097468001638907133099936.json
2021-12-07 22:00:28 CARACal.Stimela.flag__cal-rfi-ms0 INFO: Starting container [flag__cal_rfi_ms0-1401256097468001638907133099936]. Timeout set to -1. The container ID is printed below.
# running cd /scratch3/projects/fornax/data_reduction/mfs03/.stimela_workdir-16389071327985396 && singularity run --workdir /scratch3/projects/fornax/data_reduction/mfs03/.stimela_workdir-16389071327985396 --containall  --bind /scratch3/projects/fornax/data_reduction/mfs03/.stimela_workdir-16389071327985396/stimela_parameter_files/flag__cal_rfi_ms0-1401256097468001638907133099936.json:/stimela_mount/configfile:ro --bind /idia/projects/fornax/caracal-venv/lib/python3.8/site-packages/stimela/cargo/cab/autoflagger/src:/stimela_mount/code:ro --bind /scratch3/projects/fornax/data_reduction/mfs03/.stimela_workdir-16389071327985396/passwd:/etc/passwd:rw --bind /scratch3/projects/fornax/data_reduction/mfs03/.stimela_workdir-16389071327985396/group:/etc/group:rw --bind /idia/projects/fornax/caracal-venv/bin/stimela_runscript:/singularity:ro --bind /scratch3/projects/fornax/data_reduction/mfs03/msdir:/stimela_mount/msdir:rw --bind /scratch3/projects/fornax/data_reduction/mfs03/input:/stimela_mount/input:ro --bind /scratch3/projects/fornax/data_reduction/mfs03/output:/stimela_mount/output:rw --bind /scratch3/projects/fornax/data_reduction/mfs03/output/tmp:/stimela_mount/output/tmp:rw /software/astro/caracal/STIMELA_IMAGES_1.7.1/stimela_aoflagger_1.2.0.sif /singularity
# WARNING: Overriding HOME environment variable with SINGULARITYENV_HOME is not permitted
# AOFlagger 2.13.0 (2018-11-07) command line application
# This program will execute an RFI strategy as can be created with the RFI gui
# and executes it on one or several observations.
# 
# Author: André Offringa (offringa@gmail.com)
# 
# Modified single-baseline strategy so it will execute strategy on all baselines and write flags.
# Starting strategy on 2021-Dec-07 22:00:30.828620
# 0% : strategy...
# 0% : +-For each measurement set...
# 0% : +-+-Processing measurement set /stimela_mount/msdir/1638396089_sdp_l0-cal.ms...

paoloserra commented 2 years ago

With Stimela 1.7.3 on Singularity we still do not print out the AOFlagger command.

2022-02-18 17:03:17 CARACal.Stimela.flag__spw-rfi-ms0 INFO: Starting container [flag__spw_rfi_ms0-14008364558001616451964436631444]. Timeout set to -1. The container ID is printed below.
# running cd /scratch3/projects/fornax/data_reduction/mfs30/.stimela_workdir-16451964433932874 && singularity run --workdir /scratch3/projects/fornax/data_reduction/mfs30/.stimela_workdir-16451964433932874 --containall  --bind /scratch3/projects/fornax/data_reduction/mfs30/.stimela_workdir-16451964433932874/stimela_parameter_files/flag__spw_rfi_ms0-14008364558001616451964436631444.json:/stimela_mount/configfile:ro --bind /idia/projects/fornax/caracal-venv/lib/python3.8/site-packages/stimela/cargo/cab/autoflagger/src:/stimela_mount/code:ro --bind /scratch3/projects/fornax/data_reduction/mfs30/.stimela_workdir-16451964433932874/passwd:/etc/passwd:rw --bind /scratch3/projects/fornax/data_reduction/mfs30/.stimela_workdir-16451964433932874/group:/etc/group:rw --bind /idia/projects/fornax/caracal-venv/bin/stimela_runscript:/singularity:ro --bind /scratch3/projects/fornax/data_reduction/mfs30/msdir:/stimela_mount/msdir:rw --bind /scratch3/projects/fornax/data_reduction/mfs30/input:/stimela_mount/input:ro --bind /scratch3/projects/fornax/data_reduction/mfs30/output_spw0:/stimela_mount/output:rw --bind /scratch3/projects/fornax/data_reduction/mfs30/output_spw0/tmp:/stimela_mount/output/tmp:rw /software/astro/caracal/STIMELA_IMAGES_1.7.3/stimela_aoflagger_1.2.0.sif /singularity
# WARNING: Overriding HOME environment variable with SINGULARITYENV_HOME is not permitted
# AOFlagger 2.13.0 (2018-11-07) command line application
# This program will execute an RFI strategy as can be created with the RFI gui
# and executes it on one or several observations.
# 
# Author: André Offringa (offringa@gmail.com)
# 
# Modified single-baseline strategy so it will execute strategy on all baselines and write flags.
# Starting strategy on 2022-Feb-18 17:03:19.505590
# 0% : strategy...

Athanaseus commented 10 months ago

Look into this when updating to new AOflagger release

caracal-pipeline / caracal

full commands missing from log files #944