test: setup basic cli tests with pytest

kelly-sovacool commented 9 months ago

Changes

move renee script to src/renee/__main__.py, which effectively makes it a python package for compatibility with pytest.
bin/redirect (symlinked to bin/renee) now calls src/renee/__main__.py, so the interface won't be any different for users.
new tests directory for pytest.
created basic tests for help & version CLI options.

Issues

PR Checklist

(~Strikethrough~ any points that are not applicable.)

[x] This comment contains a description of changes with justifications, with any relevant issues linked.
~[ ] Update docs if there are any API changes.~
~[ ] Update CHANGELOG.md with a short description of any user-facing changes and reference the PR number. Guidelines: https://keepachangelog.com/en/1.1.0/~

kelly-sovacool commented 9 months ago

test run completed successfully on biowulf

/data/CCBR_Pipeliner/Pipelines/RENEE/renee-dev-sovacool/bin/renee run \
    --input /data/CCBR_Pipeliner/Pipelines/RENEE/develop/.tests/*.R1.fastq.gz \
    --genome hg38_30 \
    --mode slurm \
    --output /data/$USER/renee_test_dev \
    --sif-cache /data/CCBR_Pipeliner/SIFS

slsevilla commented 9 months ago

Sorry; took forever to get to this. test run deployed: /data/CCBR_Pipeliner/Pipelines/RENEE/renee-dev-sevillas2/bin/renee run \ --input /data/CCBR_Pipeliner/Pipelines/RENEE/develop/.tests/*.R1.fastq.gz \ --genome hg38_30 \ --mode slurm \ --output /data/$USER/renee_test_dev \ --sif-cache /data/CCBR_Pipeliner/SIFS

slsevilla commented 9 months ago

The pipeline itself completed successfully, but the snakemake jobby job failed Success

[Tue Feb 27 17:57:14 2024]
Finished job 0.
73 of 73 steps (100%) done
Complete log: .snakemake/log/2024-02-27T170953.228816.snakemake.log

Jobby failure

run_jobby_on_snakemake_log logfiles/snakemake.log | tee logfiles/snakemake.log.jobby | cut -f2,3,18 > logfiles/snakemake.log.jobby.short
/usr/bin/bash: run_jobby_on_snakemake_log: command not found
CalledProcessError in file /data/sevillas2/renee_test_dev/workflow/Snakefile, line 241:
Command 'set -euo pipefail;  run_jobby_on_snakemake_log logfiles/snakemake.log | tee logfiles/snakemake.log.jobby | cut -f2,3,18 > logfiles/snakemake.log.jobby.short' returned non-zero exit status 127.
  File "/data/sevillas2/renee_test_dev/workflow/Snakefile", line 241, in __onsuccess
OnSuccess
run_jobby_on_snakemake_log logfiles/snakemake.log | tee logfiles/snakemake.log.jobby | cut -f2,3,18 > logfiles/snakemake.log.jobby.short

I assume it's becuse I didn't manually 'module load ccbrpipeliner', but can you see if there's anything else I'm missing @kelly-sovacool

Complete failure log

printenv
CONDA_SHLVL=1
SLURM_MEM_PER_CPU=2048
LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
CONDA_EXE=/data/CCBR_Pipeliner/db/PipeDB/Conda/bin/conda
SRUN_DEBUG=3
SLURM_NODEID=0
SLURM_TASK_PID=244046
__LMOD_REF_COUNT_PATH=/usr/local/apps/snakemake/7.32.3/bin:1;/usr/local/current/singularity/4.0.1/bin:1;/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311/bin:1;/data/CCBR_Pipeliner/db/PipeDB/Conda/condabin:1;/data/khanlab/projects/HLA-tools/sam/hisatgenotype:2;/data/khanlab/projects/HLA-tools/sam/hisatgenotype/hisat2:2;/home/sevillas2/bin/vast-tools:2;/home/sevillas2/git/HPC_DME_APIs/utils/scripts:2;/usr/local/slurm/bin:2;/usr/local/bin:2;/usr/X11R6/bin:1;/usr/local/jdk/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/usr/local/mysql/bin:1;/data/CCBR_Pipeliner/bin/netMHC/netMHCpan-4.1:2;/data/CCBR_Pipeliner/bin/vast-tools:2;/data/CCBR_Pipeliner/db/PipeDB/bin/gh_1.7.0_linux_amd64/bin:2;/home/sevillas2/.local/bin:1;/home/sevillas2/bin:1;/gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/RENEE/renee-dev-sevillas2/bin:1
_ModuleTable002_=bmFrZW1ha2UgPSB7CmZuID0gIi91c3IvbG9jYWwvbG1vZC9tb2R1bGVmaWxlcy9zbmFrZW1ha2UvNy4zMi4zLmx1YSIsCmZ1bGxOYW1lID0gInNuYWtlbWFrZS83LjMyLjMiLApsb2FkT3JkZXIgPSAyLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gInNuYWtlbWFrZSIsCndWID0gIl4wMDAwMDAwNy4wMDAwMDAwMzIuMDAwMDAwMDAzLip6ZmluYWwiLAp9LAp9LAptcGF0aEEgPSB7CiIvdXNyL2xvY2FsL2xtb2QvbW9kdWxlZmlsZXMiLAp9LApzeXN0ZW1CYXNlTVBBVEggPSAiL3Vzci9sb2NhbC9sbW9kL21vZHVsZWZpbGVzIiwKfQo=
SSH_CONNECTION=10.243.158.212 49561 128.231.2.9 22
SLURM_PRIO_PROCESS=0
GOTO_NUM_THREADS=1
LANG=en_US.UTF-8
SLURM_SUBMIT_DIR=/gpfs/gsfs12/users/sevillas2/renee_test_dev
DISPLAY=10.2.4.187:10.0
HOSTNUM=24
HISTTIMEFORMAT=%H:%M  
JAVA_LD_LIBRARY_PATH=/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311/lib/jvm/lib/server
HOSTNAME=cn4292
OPENBLAS_NUM_THREADS=1
OLDPWD=/gpfs/gsfs12/users/sevillas2/renee_test_dev
COBBLER_SERVER=10.1.201.7
ENVIRONMENT=BATCH
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=
VECLIB_MAXIMUM_THREADS=1
LMOD_FAMILY_SINGULARITY=singularity
CONDA_PREFIX=/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311
JAVA_HOME=/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311/lib/jvm
SLURM_PROCID=0
SLURM_JOB_GID=57786
R_LIBS_USER=/data/sevillas2/R/%v
SLURMD_NODENAME=cn4292
SLURM_TASKS_PER_NODE=2
biowulf_FAMILY_SINGULARITY_VERSION=4.0.1
biowulf_FAMILY_SINGULARITY=singularity
NUMEXPR_NUM_THREADS=1
S_COLORS=auto
_CE_M=
which_declare=declare -f
XDG_SESSION_ID=204670
SLURM_NNODES=1
USER=sevillas2
SELINUX_ROLE_REQUESTED=
__LMOD_REF_COUNT_MODULEPATH=/usr/local/lmod/modulefiles:1
LMOD_FAMILY_SINGULARITY_VERSION=4.0.1
INPUTRC=/etc/inputrc
PWD=/data/sevillas2/renee_test_dev
SLURM_JOB_NODELIST=cn4292
HOME=/home/sevillas2
SLURM_CLUSTER_NAME=biowulf
CONDA_PYTHON_EXE=/data/CCBR_Pipeliner/db/PipeDB/Conda/bin/python
SLURM_NODELIST=cn4292
I2HOSTNUM=11
SSH_CLIENT=10.243.158.212 49561 22
LMOD_VERSION=8.7
SINGULARITY_CACHEDIR=/data/sevillas2/renee_test_dev/.singularity
SELINUX_LEVEL_REQUESTED=
https_proxy=http://dtn24-e0:3128
KRB5CCNAME=FILE:/tmp/krb5cc_37844_hgscGp
SLURM_JOB_CPUS_PER_NODE=2
BASH_ENV=/usr/local/lmod/lmod/8.7/init/bash
HISTFILE=/home/sevillas2/.bash_history_biowulf
SLURM_TOPOLOGY_ADDR=cn4292
http_proxy=http://dtn24-e0:3128
_CE_CONDA=
SLURM_WORKING_CLUSTER=biowulf:slurmctl:6802:9728:109
GSETTINGS_SCHEMA_DIR=/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311/share/glib-2.0/schemas
MKL_NUM_THREADS=1
SLURM_JOB_NAME=pl:renee
TMPDIR=/tmp
SQUID_PORT=3128
PVFSTAB_FILE=/usr/local/etc/pvfstab
LMOD_sys=Linux
SLURM_JOBID=20652926
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IDg2NDAwLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpzaW5ndWxhcml0eSA9ICJzaW5ndWxhcml0eSIsCn0sCm1UID0gewpzaW5ndWxhcml0eSA9IHsKZm4gPSAiL3Vzci9sb2NhbC9sbW9kL21vZHVsZWZpbGVzL3Npbmd1bGFyaXR5LzQuMC4xLmx1YSIsCmZ1bGxOYW1lID0gInNpbmd1bGFyaXR5LzQuMC4xIiwKbG9hZE9yZGVyID0gMSwKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJzaW5ndWxhcml0eSIsCndWID0gIl4wMDAwMDAwNC4wMDAwMDAwMDAuMDAwMDAwMDAxLip6ZmluYWwiLAp9LApz
SLURM_CONF=/usr/local/slurm-22.05/slurm-22.05.10/etc/slurm.conf
LOADEDMODULES=singularity/4.0.1:snakemake/7.32.3
HPC_DM_UTILS=/home/sevillas2/git/HPC_DME_APIs/utils
__LMOD_REF_COUNT_MANPATH=/usr/local/current/singularity/4.0.1/share/man:1;/usr/local/slurm/share/man:2;/usr/local/lmod/lmod/8.7/share/man:1
SLURM_NODE_ALIASES=(null)
SLURM_JOB_QOS=global
LMOD_ROOT=/usr/local/lmod/lmod
SLURM_TOPOLOGY_ADDR_PATTERN=node
CONDA_PROMPT_MODIFIER=(py311) 
SSH_TTY=/dev/pts/1969
MAIL=/var/spool/mail/sevillas2
SLURM_CPUS_ON_NODE=2
SLURM_JOB_NUM_NODES=1
TERM=xterm-256color
SHELL=/bin/bash
SLURM_JOB_UID=37844
_ModuleTable_Sz_=2
GLOBAL_ENV=set
SLURM_JOB_PARTITION=norm
SELINUX_USE_CURRENT_RANGE=
SLURM_JOB_USER=sevillas2
SHLVL=7
SLURM_SUBMIT_HOST=cn2079
PYTHONPATH=/data/khanlab/projects/HLA-tools/sam/hisatgenotype/hisatgenotype_modules:/data/khanlab/projects/HLA-tools/sam/hisatgenotype/hisatgenotype_modules:
PRINTER=unknown_printer
SLURM_JOB_ACCOUNT=malikm
MANPATH=/usr/local/current/singularity/4.0.1/share/man:/usr/local/slurm/share/man:/usr/local/lmod/lmod/8.7/share/man::
GDK_BACKEND=x11
MODULEPATH=/usr/local/lmod/modulefiles
SLURM_GTIDS=0
LOGNAME=sevillas2
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-GNsp4Dxtwb,guid=0405591185f709acde89bac865de23b3
MODULEPATH_ROOT=/usr/local/lmod/modulefiles
PATH=/usr/local/apps/snakemake/7.32.3/bin:/usr/local/current/singularity/4.0.1/bin:/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311/bin:/data/CCBR_Pipeliner/db/PipeDB/Conda/condabin:/data/khanlab/projects/HLA-tools/sam/hisatgenotype:/data/khanlab/projects/HLA-tools/sam/hisatgenotype/hisat2:/home/sevillas2/bin/vast-tools:/home/sevillas2/git/HPC_DME_APIs/utils/scripts:/usr/local/slurm/bin:/usr/local/bin:/usr/X11R6/bin:/usr/local/jdk/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/mysql/bin:/data/CCBR_Pipeliner/bin/netMHC/netMHCpan-4.1:/data/CCBR_Pipeliner/bin/vast-tools:/data/CCBR_Pipeliner/db/PipeDB/bin/gh_1.7.0_linux_amd64/bin:/home/sevillas2/.local/bin:/home/sevillas2/bin:/gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/RENEE/renee-dev-sevillas2/bin
SLURM_JOB_ID=20652926
_LMFILES_=/usr/local/lmod/modulefiles/singularity/4.0.1.lua:/usr/local/lmod/modulefiles/snakemake/7.32.3.lua
DEBUGINFOD_URLS=https://debuginfod.centos.org/ 
MODULESHOME=/usr/local/lmod/lmod/8.7
CONDA_DEFAULT_ENV=py311
LMOD_SETTARG_FULL_SUPPORT=no
HISTSIZE=1000
SLURM_JOB_CPUS_PER_NODE_PACK_GROUP_0=32
LMOD_PKG=/usr/local/lmod/lmod/8.7
HISTFILESIZE=1000
ftp_proxy=http://dtn24-e0:3128
RSYNC_PROXY=dtn24-e0:3128
LMOD_CMD=/usr/local/lmod/lmod/8.7/libexec/lmod
SLURM_LOCALID=0
OMP_NUM_THREADS=1
LESSOPEN=||/usr/bin/lesspipe.sh %s
LMOD_DIR=/usr/local/lmod/lmod/8.7/libexec
BASH_FUNC_i2proxyon%%=() {  echo "[+] Enabling I2 web proxy ...";
 export https_proxy=http://dtn${I2HOSTNUM}-e0:${SQUID_PORT}
}
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
BASH_FUNC_proxyon%%=() {  echo "[+] Enabling web proxy ...";
 export http_proxy=http://dtn${HOSTNUM}-e0:${SQUID_PORT};
 export https_proxy=http://dtn${HOSTNUM}-e0:${SQUID_PORT};
 export ftp_proxy=http://dtn${HOSTNUM}-e0:${SQUID_PORT};
 export RSYNC_PROXY=dtn${HOSTNUM}-e0:${SQUID_PORT}
}
BASH_FUNC_module%%=() {  local __lmod_my_status;
 local __lmod_sh_dbg;
 if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
 case "$-" in 
 *v*x*)
 __lmod_sh_dbg='vx'
 ;;
 *v*)
 __lmod_sh_dbg='v'
 ;;
 *x*)
 __lmod_sh_dbg='x'
 ;;
 esac;
 fi;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 set +$__lmod_sh_dbg;
 echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's output" 1>&2;
 fi;
 eval "$($LMOD_CMD bash "$@")" && eval $(${LMOD_SETTARG_CMD:-:} -s sh);
 __lmod_my_status=$?;
 if [ -n "${__lmod_sh_dbg:-}" ]; then
 echo "Shell debugging restarted" 1>&2;
 set -$__lmod_sh_dbg;
 fi;
 return $__lmod_my_status
}
BASH_FUNC_proxyoff%%=() {  echo "[-] Disabling web proxy ...";
 unset http_proxy;
 unset https_proxy;
 unset ftp_proxy;
 unset RSYNC_PROXY
}
BASH_FUNC_i2proxyoff%%=() {  echo "[-] Disabling I2 web proxy ...";
 unset https_proxy
}
BASH_FUNC_ml%%=() {  eval "$($LMOD_DIR/ml_cmd "$@")"
}
_=/data/CCBR_Pipeliner/db/PipeDB/Conda/envs/py311/bin/printenv
module list

Currently Loaded Modules:
  1) singularity/4.0.1   2) snakemake/7.32.3

run_jobby_on_snakemake_log logfiles/snakemake.log | tee logfiles/snakemake.log.jobby | cut -f2,3,18 > logfiles/snakemake.log.jobby.short
/usr/bin/bash: run_jobby_on_snakemake_log: command not found
CalledProcessError in file /data/sevillas2/renee_test_dev/workflow/Snakefile, line 241:
Command 'set -euo pipefail;  run_jobby_on_snakemake_log logfiles/snakemake.log | tee logfiles/snakemake.log.jobby | cut -f2,3,18 > logfiles/snakemake.log.jobby.short' returned non-zero exit status 127.
  File "/data/sevillas2/renee_test_dev/workflow/Snakefile", line 241, in __onsuccess
OnSuccess
run_jobby_on_snakemake_log logfiles/snakemake.log | tee logfiles/snakemake.log.jobby | cut -f2,3,18 > logfiles/snakemake.log.jobby.short

kelly-sovacool commented 9 months ago

I assume it's becuse I didn't manually 'module load ccbrpipeliner', but can you see if there's anything else I'm missing @kelly-sovacool

Right, the ccbrpipeliner module adds jobby and various helper scripts to the path (including run_jobby_on_snakemake_log)

CCBR / RENEE