search

xenon-middleware / xenon-cli

Perform files and jobs operations with Xenon library from command line
http://nlesc.github.io/Xenon/
Apache License 2.0
2 stars 3 forks source link

Failed to submit jobs via 'at' adaptor #81

Open arnikz opened 3 years ago

arnikz commented 3 years ago

Hi,

I'm running into this issue with sv-{callers,gen}/xenon workflows. Could you check the commands, perhaps I missed something? Thanks.

$ xenon --version
Xenon CLI v3.0.5, Xenon library v3.1.0, Xenon cloud library v3.0.2
$ SCH=at
$ snakemake --use-conda -j --cluster "xenon -vvv scheduler $SCH --location local:// submit --inherit-env --max-run-time 60 --working-directory ."
...
at adaptor: Unsupported maximum runtime: 60
Error submitting jobscript (exit code 1):
22:07:40.058 [main] DEBUG n.e.x.a.s.ScriptingScheduler - creating sub scheduler for at adaptor at local://
22:07:40.071 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - Creating JobQueueScheduler for Adaptor local with multiQThreads: 4 and pollingDelay: 1000

However, this arg is listed below (related to #77):

usage: xenon scheduler at --location LOCATION submit [-h] [--queue QUEUE] [--env KEY=VAL] [--inherit-env]
             [--max-run-time MAX_RUN_TIME] [--tasks TASKS] [--cores-per-task CORES_PER_TASK] [--tasks-per-node TASKS_PER_NODE]
             [--start-per-task] [--working-directory WORKING_DIRECTORY] [--max-memory MAX_MEMORY]
             [--scheduler-argument SCHEDULER_ARGUMENTS] [--temp-space TEMP_SPACE] [--stdin STDIN] [--stdout STDOUT]
             [--stderr STDERR] [--long] [--name NAME] executable [args [args ...]]

Let's leave it out from the command-line (related to #75):

$ snakemake --use-conda -j --cluster "xenon -vvv scheduler $SCH --location local:// submit --inherit-env --working-directory ."
...
local adaptor: Interactive job failed to start!
Error submitting jobscript (exit code 1):
22:09:05.641 [main] DEBUG n.e.x.a.s.ScriptingScheduler - creating sub scheduler for at adaptor at local://
22:09:05.651 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - Creating JobQueueScheduler for Adaptor local with multiQThreads: 4 and pollingDelay: 1000
22:09:05.692 [main] DEBUG n.e.x.adaptors.schedulers.at.AtUtils - Created job script:echo "#AT_JOBNAME xenon" >> /tmp/xenon.at.0
echo "#AT_WORKDIR /home/arnikz/Downloads/repos/sv-callers/snakemake/." >> /tmp/xenon.at.0
echo "#AT_STARTTIME now" >> /tmp/xenon.at.0
echo "#AT_INPUT /dev/null" >> /tmp/xenon.at.0
echo "#AT_OUTPUT /dev/null" >> /tmp/xenon.at.0
echo "#AT_ERROR /dev/null" >> /tmp/xenon.at.0
echo "#AT_ENV PATH=/home/arnikz/miniconda3/envs/wf/bin:/home/arnikz/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin" >> /tmp/xenon.at.0
echo "#AT_ENV XAUTHORITY=/home/arnikz/.Xauthority" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_DATA_DIRS=/usr/share/xfce4:/home/arnikz/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/usr/share" >> /tmp/xenon.at.0
echo "#AT_ENV GDMSESSION=lightdm-xsession" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_DEFAULT_ENV=wf" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_PYTHON_EXE=/home/arnikz/miniconda3/bin/python" >> /tmp/xenon.at.0
echo "#AT_ENV DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-q3kSbPeqvU,guid=71a438350238b6891539b7286112e378" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_PREFIX=/home/arnikz/miniconda3/envs/wf" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_CURRENT_DESKTOP=XFCE" >> /tmp/xenon.at.0
echo "#AT_ENV SSH_AGENT_PID=3237" >> /tmp/xenon.at.0
echo "#AT_ENV COLORTERM=truecolor" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_PREFIX_1=/home/arnikz/miniconda3" >> /tmp/xenon.at.0
echo "#AT_ENV SESSION_MANAGER=local/mx:@/tmp/.ICE-unix/3031,unix/mx:/tmp/.ICE-unix/3031" >> /tmp/xenon.at.0
echo "#AT_ENV LOGNAME=arnikz" >> /tmp/xenon.at.0
echo "#AT_ENV PWD=/home/arnikz/Downloads/repos/sv-callers/snakemake" >> /tmp/xenon.at.0
echo "#AT_ENV QT_PLATFORMTHEME=gtk2" >> /tmp/xenon.at.0
echo "#AT_ENV WINDOWID=90498603" >> /tmp/xenon.at.0
echo "#AT_ENV SHELL=/bin/bash" >> /tmp/xenon.at.0
echo "#AT_ENV QT_SCALE_FACTOR=1" >> /tmp/xenon.at.0
echo "#AT_ENV JAVA_HOME_CONDA_BACKUP=" >> /tmp/xenon.at.0
echo "#AT_ENV OLDPWD=/home/arnikz/miniconda3/envs/wf" >> /tmp/xenon.at.0
echo "#AT_ENV GTK_MODULES=canberra-gtk-module" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0" >> /tmp/xenon.at.0
echo "#AT_ENV PANEL_GDK_CORE_DEVICE_EVENTS=0" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_PROMPT_MODIFIER=(wf) " >> /tmp/xenon.at.0
echo "#AT_ENV LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SESSION_DESKTOP=lightdm-xsession" >> /tmp/xenon.at.0
echo "#AT_ENV SHLVL=1" >> /tmp/xenon.at.0
echo "#AT_ENV QT_PLATFORM_PLUGIN=gtk2" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_EXE=/home/arnikz/miniconda3/bin/conda" >> /tmp/xenon.at.0
echo "#AT_ENV JAVA_HOME=/home/arnikz/miniconda3/envs/wf" >> /tmp/xenon.at.0
echo "#AT_ENV TERM=xterm-256color" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_CONFIG_DIRS=/etc/xdg" >> /tmp/xenon.at.0
echo "#AT_ENV LANG=en_US.UTF-8" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SESSION_ID=1" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SESSION_TYPE=x11" >> /tmp/xenon.at.0
echo "#AT_ENV DISPLAY=:0.0" >> /tmp/xenon.at.0
echo "#AT_ENV _CE_M=" >> /tmp/xenon.at.0
echo "#AT_ENV CONDA_SHLVL=2" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SESSION_CLASS=user" >> /tmp/xenon.at.0
echo "#AT_ENV _=/home/arnikz/miniconda3/envs/wf/bin/snakemake" >> /tmp/xenon.at.0
echo "#AT_ENV JAVA_LD_LIBRARY_PATH=/home/arnikz/miniconda3/envs/wf/lib/server" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_GREETER_DATA_DIR=/var/lib/lightdm/data/arnikz" >> /tmp/xenon.at.0
echo "#AT_ENV DESKTOP_SESSION=lightdm-xsession" >> /tmp/xenon.at.0
echo "#AT_ENV USER=arnikz" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_MENU_PREFIX=xfce-" >> /tmp/xenon.at.0
echo "#AT_ENV VTE_VERSION=5402" >> /tmp/xenon.at.0
echo "#AT_ENV QT_ACCESSIBILITY=1" >> /tmp/xenon.at.0
echo "#AT_ENV JAVA_LD_LIBRARY_PATH_BACKUP=" >> /tmp/xenon.at.0
echo "#AT_ENV SSH_AUTH_SOCK=/tmp/ssh-9ZJzaq6MEFJr/agent.3031" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_SEAT=seat0" >> /tmp/xenon.at.0
echo "#AT_ENV _CE_CONDA=" >> /tmp/xenon.at.0
echo "#AT_ENV QT_QPA_PLATFORMTHEME=gtk2" >> /tmp/xenon.at.0
echo "#AT_ENV TENANT_ID=5172c716-cdc6-4db9-8382-f3c302d8f86e" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_VTNR=7" >> /tmp/xenon.at.0
echo "#AT_ENV XDG_RUNTIME_DIR=/run/user/1000" >> /tmp/xenon.at.0
echo "#AT_ENV HOME=/home/arnikz" >> /tmp/xenon.at.0
echo "#AT_EXEC /home/arnikz/Downloads/repos/sv-callers/snakemake/.snakemake/tmp.5vtl2b3_/snakejob.delly_p.13.sh" >> /tmp/xenon.at.0
export PATH="/home/arnikz/miniconda3/envs/wf/bin:/home/arnikz/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin"
export XAUTHORITY="/home/arnikz/.Xauthority"
export XDG_DATA_DIRS="/usr/share/xfce4:/home/arnikz/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/usr/share"
export GDMSESSION="lightdm-xsession"
export CONDA_DEFAULT_ENV="wf"
export CONDA_PYTHON_EXE="/home/arnikz/miniconda3/bin/python"
export DBUS_SESSION_BUS_ADDRESS="unix:abstract=/tmp/dbus-q3kSbPeqvU,guid=71a438350238b6891539b7286112e378"
export CONDA_PREFIX="/home/arnikz/miniconda3/envs/wf"
export XDG_CURRENT_DESKTOP="XFCE"
export SSH_AGENT_PID="3237"
export COLORTERM="truecolor"
export CONDA_PREFIX_1="/home/arnikz/miniconda3"
export SESSION_MANAGER="local/mx:@/tmp/.ICE-unix/3031,unix/mx:/tmp/.ICE-unix/3031"
export LOGNAME="arnikz"
export PWD="/home/arnikz/Downloads/repos/sv-callers/snakemake"
export QT_PLATFORMTHEME="gtk2"
export WINDOWID="90498603"
export SHELL="/bin/bash"
export QT_SCALE_FACTOR="1"
export JAVA_HOME_CONDA_BACKUP=""
export OLDPWD="/home/arnikz/miniconda3/envs/wf"
export GTK_MODULES="canberra-gtk-module"
export XDG_SESSION_PATH="/org/freedesktop/DisplayManager/Session0"
export PANEL_GDK_CORE_DEVICE_EVENTS="0"
export CONDA_PROMPT_MODIFIER="(wf) "
export LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:"
export XDG_SESSION_DESKTOP="lightdm-xsession"
export SHLVL="1"
export QT_PLATFORM_PLUGIN="gtk2"
export CONDA_EXE="/home/arnikz/miniconda3/bin/conda"
export JAVA_HOME="/home/arnikz/miniconda3/envs/wf"
export TERM="xterm-256color"
export XDG_CONFIG_DIRS="/etc/xdg"
export LANG="en_US.UTF-8"
export XDG_SEAT_PATH="/org/freedesktop/DisplayManager/Seat0"
export XDG_SESSION_ID="1"
export XDG_SESSION_TYPE="x11"
export DISPLAY=":0.0"
export _CE_M=""
export CONDA_SHLVL="2"
export XDG_SESSION_CLASS="user"
export _="/home/arnikz/miniconda3/envs/wf/bin/snakemake"
export JAVA_LD_LIBRARY_PATH="/home/arnikz/miniconda3/envs/wf/lib/server"
export XDG_GREETER_DATA_DIR="/var/lib/lightdm/data/arnikz"
export DESKTOP_SESSION="lightdm-xsession"
export USER="arnikz"
export XDG_MENU_PREFIX="xfce-"
export VTE_VERSION="5402"
export QT_ACCESSIBILITY="1"
export JAVA_LD_LIBRARY_PATH_BACKUP=""
export SSH_AUTH_SOCK="/tmp/ssh-9ZJzaq6MEFJr/agent.3031"
export XDG_SEAT="seat0"
export _CE_CONDA=""
export QT_QPA_PLATFORMTHEME="gtk2"
export TENANT_ID="5172c716-cdc6-4db9-8382-f3c302d8f86e"
export XDG_VTNR="7"
export XDG_RUNTIME_DIR="/run/user/1000"
export HOME="/home/arnikz"
cd '/home/arnikz/Downloads/repos/sv-callers/snakemake/.' && /home/arnikz/Downloads/repos/sv-callers/snakemake/.snakemake/tmp.5vtl2b3_/snakejob.delly_p.13.sh < '/dev/null' > '/dev/null' 2> '/dev/null' &
PID=$!
echo "#AT_PID $PID" >> /tmp/xenon.at.0
wait $PID
EXIT_CODE=$?
echo "#AT_EXIT $EXIT_CODE" >> /tmp/xenon.at.0
 from description JobDescription [name=null, queueName=null, executable=/home/arnikz/Downloads/repos/sv-callers/snakemake/.snakemake/tmp.5vtl2b3_/snakejob.delly_p.13.sh, arguments=[], schedulerArguments=[], stdin=null, stdout=null, stderr=null, workingDirectory=., environment={PATH=/home/arnikz/miniconda3/envs/wf/bin:/home/arnikz/miniconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin, XAUTHORITY=/home/arnikz/.Xauthority, XDG_DATA_DIRS=/usr/share/xfce4:/home/arnikz/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/usr/share, GDMSESSION=lightdm-xsession, CONDA_DEFAULT_ENV=wf, CONDA_PYTHON_EXE=/home/arnikz/miniconda3/bin/python, DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-q3kSbPeqvU,guid=71a438350238b6891539b7286112e378, CONDA_PREFIX=/home/arnikz/miniconda3/envs/wf, XDG_CURRENT_DESKTOP=XFCE, SSH_AGENT_PID=3237, COLORTERM=truecolor, CONDA_PREFIX_1=/home/arnikz/miniconda3, SESSION_MANAGER=local/mx:@/tmp/.ICE-unix/3031,unix/mx:/tmp/.ICE-unix/3031, LOGNAME=arnikz, PWD=/home/arnikz/Downloads/repos/sv-callers/snakemake, QT_PLATFORMTHEME=gtk2, WINDOWID=90498603, SHELL=/bin/bash, QT_SCALE_FACTOR=1, JAVA_HOME_CONDA_BACKUP=, OLDPWD=/home/arnikz/miniconda3/envs/wf, GTK_MODULES=canberra-gtk-module, XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0, PANEL_GDK_CORE_DEVICE_EVENTS=0, CONDA_PROMPT_MODIFIER=(wf) , LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:, XDG_SESSION_DESKTOP=lightdm-xsession, SHLVL=1, QT_PLATFORM_PLUGIN=gtk2, CONDA_EXE=/home/arnikz/miniconda3/bin/conda, JAVA_HOME=/home/arnikz/miniconda3/envs/wf, TERM=xterm-256color, XDG_CONFIG_DIRS=/etc/xdg, LANG=en_US.UTF-8, XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0, XDG_SESSION_ID=1, XDG_SESSION_TYPE=x11, DISPLAY=:0.0, _CE_M=, CONDA_SHLVL=2, XDG_SESSION_CLASS=user, _=/home/arnikz/miniconda3/envs/wf/bin/snakemake, JAVA_LD_LIBRARY_PATH=/home/arnikz/miniconda3/envs/wf/lib/server, XDG_GREETER_DATA_DIR=/var/lib/lightdm/data/arnikz, DESKTOP_SESSION=lightdm-xsession, USER=arnikz, XDG_MENU_PREFIX=xfce-, VTE_VERSION=5402, QT_ACCESSIBILITY=1, JAVA_LD_LIBRARY_PATH_BACKUP=, SSH_AUTH_SOCK=/tmp/ssh-9ZJzaq6MEFJr/agent.3031, XDG_SEAT=seat0, _CE_CONDA=, QT_QPA_PLATFORMTHEME=gtk2, TENANT_ID=5172c716-cdc6-4db9-8382-f3c302d8f86e, XDG_VTNR=7, XDG_RUNTIME_DIR=/run/user/1000, HOME=/home/arnikz}, tasks=1, coresPerTask=1, tasksPerNode=-1, maxMemory=-1, tempSpace=-1, startPerTask=false, maxTime=-1]
22:09:05.692 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: Submitting job
22:09:05.696 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: Created Job local-0
22:09:05.697 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: Submitting job to queue unlimited
22:09:05.699 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: Waiting for interactive job to start.
22:09:05.704 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: cleanupJob for job local-0
22:09:05.704 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: cleanupJob for job local-0
22:09:05.704 [main] DEBUG n.e.x.a.schedulers.JobQueueScheduler - local: cleanupJob for job local-0
arnikz commented 3 years ago

Interestingly, I can't reproduce the issue using another workflow. For this one, all jobs completed successfully :smile:

$ snakemake --use-conda -j --cluster "xenon -vvv scheduler $SCH --location local:// submit --inherit-env --working-directory ."
...
Tue Aug 24 05:51:28 2021]
rule copy_fastq:
    input: input/fastq/sample1/R_2.fastq.bz2
    output: output/sample1/R_2.fastq.bz2
    log: logs/copy/sample1/R_2_fastq.bz2.log
    jobid: 6
    wildcards: prefix=sample1/R, pe_reads=2, suffix=fastq.bz2
    resources: tmpdir=/tmp

Submitted job 6 with external jobid '05:51:29.217 [main] DEBUG n.e.x.a.s.ScriptingScheduler - creating sub scheduler for at adaptor at local://'.

However, I can't retrieve job accounting info:

xenon --json scheduler $SCH --location local:// list --identifier 6
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.gson.internal.bind.ReflectiveTypeAdapterFactory (file:/home/arnikz/miniconda3/envs/snakemake/lib/xenon-cli-3.0.5.jar) to field java.lang.Throwable.detailMessage
WARNING: Please consider reporting this to the maintainers of com.google.gson.internal.bind.ReflectiveTypeAdapterFactory
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
{
  "statuses": [
    {
      "jobIdentifier": "6",
      "state": "UNKNOWN",
      "exception": {
        "adaptorName": "at",
        "detailMessage": "Job 6 could not be found!",
        "stackTrace": [],
        "suppressedExceptions": []
      },
      "running": false,
      "done": false
    }
  ]
}
arnikz commented 3 years ago

I found the bugger - it's related to the workflow itself when using more cores than available. Limiting the number of cores using e.g., snakemake -j 1 ... solves the issue above.

jmaassen commented 3 years ago

Good to hear you found the issue. Are you also able to retrieve the job status if the job runs successfully? If not we can look into that issue.

I'll also have a look at the error reporting when you request more cores than available. If would be good if it was clear why the job failed.

arnikz commented 3 years ago

Are you also able to retrieve the job status if the job runs successfully? If not we can look into that issue.

Given the example above, my job completed successfully but the job status remains UNKNOWN. In addition, it would be nice to see the runtime, (peak) memory use etc. as reported for the other schedulers.

arnikz commented 3 years ago

Shall we close this job submission issue and open a new one for the job accounting?

jmaassen commented 3 years ago

I would have to look into getting the stats of the job. It would probably require some external application to provide these (like "time").

The job status should not be UNKNOWN. I'll have a look.

jmaassen commented 3 years ago

As it turns out, at itself was missing ;-)

We should check this and give a proper error message if this is the case.