search

vatlab / sos-pbs

PBS task engine for SoS
BSD 3-Clause "New" or "Revised" License
1 stars 1 forks source link

Suddenly stopped working with slurm #8

Open pgcudahy opened 2 years ago

pgcudahy commented 2 years ago

I've been using sos a lot in the past week on my university cluster but today it has suddenly stopped working with a strange environment issue. None of my steps which worked previously work anymore. For an example, here's my step to run gubbins on a multi-fasta alignment.

%sosrun gubbins -q yale_hpc_task_spooler \
-r yale_hpc_task_spooler mem="4GB" cores=2 walltime="04:00:00" nodes=1

[gubbins]
input: output_from("generate_ska_alignment")["ska_alignment"], group_by="all"
output: tree = f'/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/\
Taiwan_MKansasii.final_tree.tre',
filtered_multifasta = f'/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/\
Taiwan_MKansasii.filtered_polymorphic_sites.fasta',
recombination_gff = f'/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/\
Taiwan_MKansasii.recombination_predictions.gff'
task: walltime='06:00:00', mem="24GB", cores=8, nodes=1,
    workdir='/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'

run: expand = True
    module load miniconda
    conda activate gubbins
    cd {_output:d}
    run_gubbins.py \
        -p Taiwan_MKansasii \
        --threads 8 \
        {_input}

But it fails with this error

t025f4ebfe0a4e5d4   failed

Created 19 min ago
Started 19 min ago
Ran for 0s
TASK:
=====
run(fr"""module load miniconda
conda activate gubbins
cd {_output:d}
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 4 \
    {_input}

""")

TAGS:
=====
gubbins pgc29 w49339814024d0f8f

ENVIRONMENT:
============
__signature_vars__    {'_input', '_output', 'run'}
_depends              []
_index                0
_input                [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln')]
_output               [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.final_tree.tre'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.filtered_polymorphic_sites.fasta'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.recombination_predictions.gff')]
_runtime              {'cores': 8,
 'localhost': {'paths': {'home': '/home/pgc29/',
                         'scratch': '/home/pgc29/scratch60/'}},
 'max_cores': 200,
 'max_mem': 1200000000000,
 'max_walltime': '72:00:00',
 'mem': 24000000000,
 'nodes': 1,
 'queue': 'yale_hpc_task_spooler',
 'run_mode': 'run',
 'sig_mode': 'default',
 'verbosity': 2,
 'walltime': '06:00:00',
 'workdir': '/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'}
step_name             'gubbins'
workflow_id           'w49339814024d0f8f'

EXECUTION STATS:
================

execution script:
================
#!/bin/bash
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=3G
#SBATCH --job-name=t025f4ebfe0a4e5d4
#SBATCH --output=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.out
#SBATCH --error=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.err
#SBATCH --requeue
#SBATCH --partition=general,scavenge
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
/home/pgc29/.local/bin/sos execute t025f4ebfe0a4e5d4 -v 2 -s default -m run

standard error:
================
INFO: t025f4ebfe0a4e5d4 started
module () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
ml () {  eval $($LMOD_DIR/ml_cmd "$@")
}
# -*- shell-script -*-
#
########################################################################
# Start Lmod BASHENV
########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
   case "$-" in
     *v*x*) __lmod_vx='vx' ;;
     *v*)   __lmod_vx='v'  ;;
     *x*)   __lmod_vx='x'  ;;
   esac;
fi

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx 
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted

########################################################################
# End Lmod BASHENV
########################################################################
#
# Local Variables:
# mode: shell-script
# indent-tabs-mode: nil
# End:
module load miniconda
Lmod has detected the following error: The following module(s) are unknown:
"miniconda"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "miniconda"

Also make sure that all modulefiles written in TCL start with the string
#%Module

false
Task t025f4ebfe0a4e5d4 exits with code 1

The important bit is Lmod has detected the following error: The following module(s) are unknown: "miniconda". But when I submit the job manually (using sbatch) the following script runs fine

#!/bin/bash
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=3G
#SBATCH --job-name=gubbins
#SBATCH --output=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.out
#SBATCH --error=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.err
#SBATCH --requeue
#SBATCH --partition=general,scavenge
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
module load miniconda
conda activate gubbins
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 8 \
    /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln

This way lmod finds the miniconda module just fine and runs. My sos step was working on the cluster a couple of days ago. Now all of my sos scripts are failing with lmod not finding any modules, so it doesn't seem to be a module specific problem, but lmod itself. What could cause this?

BoPeng commented 2 years ago

It is very difficult to test what is going on here. However, judging from the error message, it looks like the computing node does not have this module. Can you find the job id of the failed job, get the node on which the task is executed, ssh to the node, and check module load runs over there?

pgcudahy commented 2 years ago

Yes, when I ssh into the node, lmod loads miniconda without issue. I've tried other sos scripts that previously worked and use common modules like GATK, trimmomatic, BWA, etc. and in every case the job fails with lmod not finding the module.

[pgc29@c23n08 ~]$ sacct -j 23712498 --format=JobID,JobName,MaxRSS,Elapsed,NodeList
JobID           JobName     MaxRSS    Elapsed        NodeList 
------------ ---------- ---------- ---------- --------------- 
23712498     t025f4ebf+              00:00:04          c18n01 
23712498.ba+      batch      1488K   00:00:04          c18n01 
23712498.ex+     extern      1076K   00:00:04          c18n01 
[pgc29@c23n08 ~]$ ssh c18n01
Warning: Permanently added 'c18n01,10.178.118.1' (ECDSA) to the list of known hosts.
[pgc29@c18n01 ~]$ module load miniconda
[pgc29@c18n01 ~]$ conda
usage: conda [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.
BoPeng commented 2 years ago

ok, on the computing node can you run the script directly?

cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
/home/pgc29/.local/bin/sos execute t025f4ebfe0a4e5d4 -v 2 -s default -m run

The module load command is supposed to be executed from here.

I need to run an errand now but the next thing you can try is to print the env inside the task script. Maybe slurm needs an option to pass all env into the script (which for PBS is -v so you may need the #SBATCH version of $PBS -v). Since it worked previously, so the admin might have changed some system settings.

Just my guess.

pgcudahy commented 2 years ago 
[pgc29@c18n01 gubbins]$ /home/pgc29/.local/bin/sos execute t025f4ebfe0a4e5d4 -v 2 -s default -m run
INFO: t025f4ebfe0a4e5d4 started

It seems to be running fine when started this way. When I use another terminal to log onto the cluster I see

[pgc29@c23n08 ~]$ sos status t025f4ebfe0a4e5d4 -v4 
t025f4ebfe0a4e5d4   running

Created 1 hr 30 min ago
Started 43 sec ago
Ran for 43 sec
TASK:
=====
run(fr"""module load miniconda
conda activate gubbins
cd {_output:d}
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 4 \
    {_input}

""")

TAGS:
=====
gubbins pgc29 w49339814024d0f8f

ENVIRONMENT:
============
__signature_vars__    {'_output', 'run', '_input'}
_depends              []
_index                0
_input                [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln')]
_output               [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.final_tree.tre'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.filtered_polymorphic_sites.fasta'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.recombination_predictions.gff')]
_runtime              {'cores': 8,
 'localhost': {'paths': {'home': '/home/pgc29/',
                         'scratch': '/home/pgc29/scratch60/'}},
 'max_cores': 200,
 'max_mem': 1200000000000,
 'max_walltime': '72:00:00',
 'mem': 24000000000,
 'nodes': 1,
 'queue': 'yale_hpc_task_spooler',
 'run_mode': 'run',
 'sig_mode': 'default',
 'verbosity': 2,
 'walltime': '06:00:00',
 'workdir': '/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'}
step_name             'gubbins'
workflow_id           'w49339814024d0f8f'

t025f4ebfe0a4e5d4.soserr:
=========================
module () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
ml () {  eval $($LMOD_DIR/ml_cmd "$@")
}
# -*- shell-script -*-
#
########################################################################
# Start Lmod BASHENV
########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
   case "$-" in
     *v*x*) __lmod_vx='vx' ;;
     *v*)   __lmod_vx='v'  ;;
     *x*)   __lmod_vx='x'  ;;
   esac;
fi

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx 
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted

########################################################################
# End Lmod BASHENV
########################################################################
#
# Local Variables:
# mode: shell-script
# indent-tabs-mode: nil
# End:
module load miniconda
EBDEVELMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0/easybuild/miniconda-4.11.0-easybuild-devel; export EBDEVELMINICONDA; EBROOTMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0; export EBROOTMINICONDA; EBVERSIONMINICONDA=4.11.0; export EBVERSIONMINICONDA; LOADEDMODULES=StdEnv:settarg:miniconda/4.11.0; export LOADEDMODULES; MANPATH=/ysm-gpfs/apps/software/miniconda/4.11.0/share/man:/ysm-gpfs/apps/software/miniconda/4.11.0/man:/usr/share/lmod/lmod/share/man::; export MANPATH; MODULEPATH=/gpfs/ysm/apps/modules/base:/gpfs/ysm/apps/modules/bio:/gpfs/ysm/apps/modules/cae:/gpfs/ysm/apps/modules/chem:/gpfs/ysm/apps/modules/compiler:/gpfs/ysm/apps/modules/data:/gpfs/ysm/apps/modules/debugger:/gpfs/ysm/apps/modules/devel:/gpfs/ysm/apps/modules/geo:/gpfs/ysm/apps/modules/ide:/gpfs/ysm/apps/modules/lang:/gpfs/ysm/apps/modules/lib:/gpfs/ysm/apps/modules/math:/gpfs/ysm/apps/modules/mpi:/gpfs/ysm/apps/modules/numlib:/gpfs/ysm/apps/modules/perf:/gpfs/ysm/apps/modules/phys:/gpfs/ysm/apps/modules/system:/gpfs/ysm/apps/modules/toolchain:/gpfs/ysm/apps/modules/tools:/gpfs/ysm/apps/modules/vis:/gpfs/ysm/apps/modules/restricted:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core; export MODULEPATH; __LMOD_REF_COUNT_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/bin:1\;/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:1\;OBJ/_x86_64_06_avx2:1\;/home/pgc29/byobu/bin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/bin:1\;/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:1\;/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Java/1.8.0_261:1\;/ysm-gpfs/apps/software/Java/1.8.0_261/bin:1\;/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:1\;/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:1\;/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:1\;/gpfs/ysm/bin:1\;/opt/TurboVNC/bin:1\;/usr/lib64/qt-3.3/bin:1\;/usr/local/bin:1\;/usr/bin:1\;/usr/local/sbin:1\;/usr/sbin:1\;/home/pgc29/.local/bin:2\;/home/pgc29/bin:2; export __LMOD_REF_COUNT_PATH; PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin; export PATH; PKG_CONFIG_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0/lib/pkgconfig; export PKG_CONFIG_PATH; PYTHONNOUSERSITE=True; export PYTHONNOUSERSITE; _LMFILES_=/etc/modulefiles/StdEnv.lua:/usr/share/lmod/lmod/modulefiles/Core/settarg.lua:/gpfs/ysm/apps/modules/tools/miniconda/4.11.0.lua; export _LMFILES_; _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT0wLjAwMTExNjAzNzM2ODc3NDQsZGVwdGhUPXt9LGZhbWlseT17fSxtVD17U3RkRW52PXtbImZuIl09Ii9ldGMvbW9kdWxlZmlsZXMvU3RkRW52Lmx1YSIsWyJmdWxsTmFtZSJdPSJTdGRFbnYiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17bG1vZD17WyJzdGlja3kiXT0xLH0sfSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJTdGRFbnYiLFsid1YiXT0iTS4qemZpbmFsIix9LG1pbmljb25kYT17WyJmbiJdPSIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Rvb2xzL21pbmljb25kYS80LjExLjAubHVhIixbImZ1bGxO; export _ModuleTable001_; _ModuleTable002_=YW1lIl09Im1pbmljb25kYS80LjExLjAiLFsibG9hZE9yZGVyIl09Myxwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJtaW5pY29uZGEiLFsid1YiXT0iMDAwMDAwMDA0LjAwMDAwMDAxMS4qemZpbmFsIix9LHNldHRhcmc9e1siZm4iXT0iL3Vzci9zaGFyZS9sbW9kL2xtb2QvbW9kdWxlZmlsZXMvQ29yZS9zZXR0YXJnLmx1YSIsWyJmdWxsTmFtZSJdPSJzZXR0YXJnIixbImxvYWRPcmRlciJdPTIscHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0ic2V0dGFyZyIsWyJ3ViJdPSJNLip6ZmluYWwiLH0sfSxtcGF0aEE9eyIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2Jh; export _ModuleTable002_; _ModuleTable003_=c2UiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2JpbyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvY2FlIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9jaGVtIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9jb21waWxlciIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvZGF0YSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvZGVidWdnZXIiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2RldmVsIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9nZW8iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2lkZSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvbGFuZyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvbGliIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9tYXRoIiwiL2dwZnMveXNtL2FwcHMv; export _ModuleTable003_; _ModuleTable004_=bW9kdWxlcy9tcGkiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL251bWxpYiIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvcGVyZiIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvcGh5cyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvc3lzdGVtIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy90b29sY2hhaW4iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Rvb2xzIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy92aXMiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Jlc3RyaWN0ZWQiLCIvZXRjL21vZHVsZWZpbGVzIiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcyIsIi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvTGludXgiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0NvcmUiLCIvdXNyL3NoYXJl; export _ModuleTable004_; _ModuleTable005_=L2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIix9LFsic3lzdGVtQmFzZU1QQVRIIl09Ii9ldGMvbW9kdWxlZmlsZXM6L3Vzci9zaGFyZS9tb2R1bGVmaWxlczovdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4Oi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvQ29yZTovdXNyL3NoYXJlL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIix9; export _ModuleTable005_; _ModuleTable_Sz_=5; export _ModuleTable_Sz_; conda deactivate; unset CONDA_EXE; unset _CE_CONDA; unset _CE_M; unset -f __conda_activate; unset -f __conda_reactivate; unset -f __conda_hashr; unset CONDA_SHLVL; unset _CONDA_EXE; unset _CONDA_ROOT; unset -f conda; source /ysm-gpfs/apps/software/miniconda/4.11.0/etc/profile.d/conda.sh;

CommandNotFoundError: Your shell has not been properly configured to use 'conda deactivate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'

# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause

__add_sys_prefix_to_path() {
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
        SYSP=$(\dirname "${CONDA_EXE}")
    else
        SYSP=$(\dirname "${CONDA_EXE}")
        SYSP=$(\dirname "${SYSP}")
    fi

    if [ -n "${WINDIR+x}" ]; then
        PATH="${SYSP}/bin:${PATH}"
        PATH="${SYSP}/Scripts:${PATH}"
        PATH="${SYSP}/Library/bin:${PATH}"
        PATH="${SYSP}/Library/usr/bin:${PATH}"
        PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
        PATH="${SYSP}:${PATH}"
    else
        PATH="${SYSP}/bin:${PATH}"
    fi
    \export PATH
}

__conda_exe() (
    __add_sys_prefix_to_path
    "$CONDA_EXE" $_CE_M $_CE_CONDA "$@"
)

__conda_hashr() {
    if [ -n "${ZSH_VERSION:+x}" ]; then
        \rehash
    elif [ -n "${POSH_VERSION:+x}" ]; then
        :  # pass
    else
        \hash -r
    fi
}

__conda_activate() {
    if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
        # Handle transition from shell activated with conda <= 4.3 to a subsequent activation
        # after conda updated to >= 4.4. See issue #6173.
        PS1="$CONDA_PS1_BACKUP"
        \unset CONDA_PS1_BACKUP
    fi
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$@")" || \return
    \eval "$ask_conda"
    __conda_hashr
}

__conda_reactivate() {
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix reactivate)" || \return
    \eval "$ask_conda"
    __conda_hashr
}

conda() {
    \local cmd="${1-__missing__}"
    case "$cmd" in
        activate|deactivate)
            __conda_activate "$@"
            ;;
        install|update|upgrade|remove|uninstall)
            __conda_exe "$@" || \return
            __conda_reactivate
            ;;
        *)
            __conda_exe "$@"
            ;;
    esac
}

if [ -z "${CONDA_SHLVL+x}" ]; then
    \export CONDA_SHLVL=0
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA:+x}" ] && [ -n "${WINDIR+x}" ]; then
        PATH="$(\dirname "$CONDA_EXE")/condabin${PATH:+":${PATH}"}"
    else
        PATH="$(\dirname "$(\dirname "$CONDA_EXE")")/condabin${PATH:+":${PATH}"}"
    fi
    \export PATH

    # We're not allowing PS1 to be unbound. It must at least be set.
    # However, we're not exporting it, which can cause problems when starting a second shell
    # via a first shell (i.e. starting zsh from bash).
    if [ -z "${PS1+x}" ]; then
        PS1=
    fi
fi
PATH='/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'; export PATH; TARG='OBJ/_x86_64_06_avx2'; export TARG; unset 'TARG_BUILD_SCENARIO'; unset 'TARG_COMPILER_FAMILY'; unset 'TARG_EXTRA'; TARG_HOST='farnam'; export TARG_HOST; TARG_MACH='x86_64_06_avx2'; export TARG_MACH; TARG_MACH_DESCRIPT='Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz'; export TARG_MACH_DESCRIPT; unset 'TARG_MPI_FAMILY'; TARG_OS='Linux-3.10.0-1160.42.2.el7.x86_64'; export TARG_OS; TARG_OS_FAMILY='Linux'; export TARG_OS_FAMILY; TARG_SUMMARY='x86_64_06_avx2'; export TARG_SUMMARY; unset 'TARG_TITLE_BAR'; TARG_TITLE_BAR_PAREN=' '; export TARG_TITLE_BAR_PAREN;
conda activate gubbins
PS1='(gubbins)'
export PATH='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'
export CONDA_PREFIX='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_SHLVL='1'
export CONDA_DEFAULT_ENV='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_PROMPT_MODIFIER='(gubbins)'
export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 4 \
    /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln

And I see output files starting to appear where they should.

BoPeng commented 2 years ago

So this means some env was not passed to the script correctly. The next step would be creating a shell script with sos execute, use sbatch to submit the job. If the job fails, maybe add an option --export=ALL.

--export=ALLDefault mode if --export is not specified. All of the user's environment will be loaded (either from the caller's environment or from a clean environment if --get-user-env is specified).

pgcudahy commented 2 years ago

That works too!? I used the script

#!/bin/bash
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=3G
#SBATCH --job-name=gubbins
#SBATCH --output=/home/pgc29/gubbins.out
#SBATCH --error=/home/pgc29/gubbins.err
#SBATCH --requeue
#SBATCH --partition=general,scavenge
#SBATCH --export=ALL
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
/home/pgc29/.local/bin/sos execute t54475dda19ec2b2d -v 2 -s default -m run

And after sbatch it's happily working away. Unfortunately I just got a text from my ISP that my service is going down for the next few hours. I'm in a remote area on a wireless microwave connection so this happens. It's also late my time so I won't be able to respond again until tomorrow. Thanks again for you help with my esoteric issues. I really appreciate it.

pgcudahy commented 2 years ago

In case this helps 

[pgc29@c16n03 ~]$ sos status t54475dda19ec2b2d -v4
t54475dda19ec2b2d   running

Created 36 min ago
Started 5 min ago
Ran for 5 min
TASK:
=====
run(fr"""module load miniconda
conda activate gubbins
cd {_output:d}
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 8 \
    {_input}

""")

TAGS:
=====
gubbins pgc29 w9142b9731f79a5e8

ENVIRONMENT:
============
__signature_vars__    {'_output', 'run', '_input'}
_depends              []
_index                0
_input                [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln')]
_output               [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.final_tree.tre'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.filtered_polymorphic_sites.fasta'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.recombination_predictions.gff')]
_runtime              {'cores': 8,
 'localhost': {'paths': {'home': '/home/pgc29/',
                         'scratch': '/home/pgc29/scratch60/'}},
 'max_cores': 200,
 'max_mem': 1200000000000,
 'max_walltime': '72:00:00',
 'mem': 24000000000,
 'nodes': 1,
 'queue': 'yale_hpc_task_spooler',
 'run_mode': 'run',
 'sig_mode': 'default',
 'verbosity': 2,
 'walltime': '06:00:00',
 'workdir': '/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'}
step_name             'gubbins'
workflow_id           'w9142b9731f79a5e8'

t54475dda19ec2b2d.soserr:
=========================
module () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
ml () {  eval $($LMOD_DIR/ml_cmd "$@")
}
# -*- shell-script -*-
#
########################################################################
# Start Lmod BASHENV
########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
   case "$-" in
     *v*x*) __lmod_vx='vx' ;;
     *v*)   __lmod_vx='v'  ;;
     *x*)   __lmod_vx='x'  ;;
   esac;
fi

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx 
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted

########################################################################
# End Lmod BASHENV
########################################################################
#
# Local Variables:
# mode: shell-script
# indent-tabs-mode: nil
# End:
module load miniconda
EBDEVELMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0/easybuild/miniconda-4.11.0-easybuild-devel; export EBDEVELMINICONDA; EBROOTMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0; export EBROOTMINICONDA; EBVERSIONMINICONDA=4.11.0; export EBVERSIONMINICONDA; LOADEDMODULES=StdEnv:settarg:miniconda/4.11.0; export LOADEDMODULES; MANPATH=/ysm-gpfs/apps/software/miniconda/4.11.0/share/man:/ysm-gpfs/apps/software/miniconda/4.11.0/man:/usr/share/lmod/lmod/share/man::; export MANPATH; MODULEPATH=/gpfs/ysm/apps/modules/base:/gpfs/ysm/apps/modules/bio:/gpfs/ysm/apps/modules/cae:/gpfs/ysm/apps/modules/chem:/gpfs/ysm/apps/modules/compiler:/gpfs/ysm/apps/modules/data:/gpfs/ysm/apps/modules/debugger:/gpfs/ysm/apps/modules/devel:/gpfs/ysm/apps/modules/geo:/gpfs/ysm/apps/modules/ide:/gpfs/ysm/apps/modules/lang:/gpfs/ysm/apps/modules/lib:/gpfs/ysm/apps/modules/math:/gpfs/ysm/apps/modules/mpi:/gpfs/ysm/apps/modules/numlib:/gpfs/ysm/apps/modules/perf:/gpfs/ysm/apps/modules/phys:/gpfs/ysm/apps/modules/system:/gpfs/ysm/apps/modules/toolchain:/gpfs/ysm/apps/modules/tools:/gpfs/ysm/apps/modules/vis:/gpfs/ysm/apps/modules/restricted:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core; export MODULEPATH; __LMOD_REF_COUNT_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/bin:1\;/home/pgc29/byobu/bin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/bin:1\;/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:1\;/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Java/1.8.0_261:1\;/ysm-gpfs/apps/software/Java/1.8.0_261/bin:1\;/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:1\;/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:1\;/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:1\;/gpfs/ysm/bin:1\;/opt/TurboVNC/bin:1\;/usr/lib64/qt-3.3/bin:1\;/usr/local/bin:1\;/usr/bin:1\;/usr/local/sbin:1\;/usr/sbin:1\;/home/pgc29/.local/bin:1\;/home/pgc29/bin:1; export __LMOD_REF_COUNT_PATH; PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin; export PATH; PKG_CONFIG_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0/lib/pkgconfig; export PKG_CONFIG_PATH; PYTHONNOUSERSITE=True; export PYTHONNOUSERSITE; _LMFILES_=/etc/modulefiles/StdEnv.lua:/usr/share/lmod/lmod/modulefiles/Core/settarg.lua:/gpfs/ysm/apps/modules/tools/miniconda/4.11.0.lua; export _LMFILES_; _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPTcyMDAsWyJjX3Nob3J0VGltZSJdPTAuMDAxMTE2MDM3MzY4Nzc0NCxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXtTdGRFbnY9e1siZm4iXT0iL2V0Yy9tb2R1bGVmaWxlcy9TdGRFbnYubHVhIixbImZ1bGxOYW1lIl09IlN0ZEVudiIsWyJsb2FkT3JkZXIiXT0xLHByb3BUPXtsbW9kPXtbInN0aWNreSJdPTEsfSx9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09IlN0ZEVudiIsWyJ3ViJdPSJNLip6ZmluYWwiLH0sbWluaWNvbmRhPXtbImZuIl09Ii9ncGZzL3lzbS9hcHBzL21vZHVsZXMvdG9vbHMvbWluaWNvbmRhLzQuMTEuMC5sdWEiLFsiZnVsbE5h; export _ModuleTable001_; _ModuleTable002_=bWUiXT0ibWluaWNvbmRhLzQuMTEuMCIsWyJsb2FkT3JkZXIiXT0zLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09Im1pbmljb25kYSIsWyJ3ViJdPSIwMDAwMDAwMDQuMDAwMDAwMDExLip6ZmluYWwiLH0sc2V0dGFyZz17WyJmbiJdPSIvdXNyL3NoYXJlL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlL3NldHRhcmcubHVhIixbImZ1bGxOYW1lIl09InNldHRhcmciLFsibG9hZE9yZGVyIl09Mixwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJzZXR0YXJnIixbIndWIl09Ik0uKnpmaW5hbCIsfSx9LG1wYXRoQT17Ii9ncGZzL3lzbS9hcHBzL21vZHVsZXMvYmFz; export _ModuleTable002_; _ModuleTable003_=ZSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvYmlvIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9jYWUiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2NoZW0iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2NvbXBpbGVyIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9kYXRhIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9kZWJ1Z2dlciIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvZGV2ZWwiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2dlbyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvaWRlIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9sYW5nIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9saWIiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL21hdGgiLCIvZ3Bmcy95c20vYXBwcy9t; export _ModuleTable003_; _ModuleTable004_=b2R1bGVzL21waSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvbnVtbGliIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9wZXJmIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9waHlzIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9zeXN0ZW0iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Rvb2xjaGFpbiIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvdG9vbHMiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3ZpcyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvcmVzdHJpY3RlZCIsIi9ldGMvbW9kdWxlZmlsZXMiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzIiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9MaW51eCIsIi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvQ29yZSIsIi91c3Ivc2hhcmUv; export _ModuleTable004_; _ModuleTable005_=bG1vZC9sbW9kL21vZHVsZWZpbGVzL0NvcmUiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL2V0Yy9tb2R1bGVmaWxlczovdXNyL3NoYXJlL21vZHVsZWZpbGVzOi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvTGludXg6L3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlOi91c3Ivc2hhcmUvbG1vZC9sbW9kL21vZHVsZWZpbGVzL0NvcmUiLH0=; export _ModuleTable005_; _ModuleTable_Sz_=5; export _ModuleTable_Sz_; conda deactivate; unset CONDA_EXE; unset _CE_CONDA; unset _CE_M; unset -f __conda_activate; unset -f __conda_reactivate; unset -f __conda_hashr; unset CONDA_SHLVL; unset _CONDA_EXE; unset _CONDA_ROOT; unset -f conda; source /ysm-gpfs/apps/software/miniconda/4.11.0/etc/profile.d/conda.sh;

CommandNotFoundError: Your shell has not been properly configured to use 'conda deactivate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'

# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause

__add_sys_prefix_to_path() {
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
        SYSP=$(\dirname "${CONDA_EXE}")
    else
        SYSP=$(\dirname "${CONDA_EXE}")
        SYSP=$(\dirname "${SYSP}")
    fi

    if [ -n "${WINDIR+x}" ]; then
        PATH="${SYSP}/bin:${PATH}"
        PATH="${SYSP}/Scripts:${PATH}"
        PATH="${SYSP}/Library/bin:${PATH}"
        PATH="${SYSP}/Library/usr/bin:${PATH}"
        PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
        PATH="${SYSP}:${PATH}"
    else
        PATH="${SYSP}/bin:${PATH}"
    fi
    \export PATH
}

__conda_exe() (
    __add_sys_prefix_to_path
    "$CONDA_EXE" $_CE_M $_CE_CONDA "$@"
)

__conda_hashr() {
    if [ -n "${ZSH_VERSION:+x}" ]; then
        \rehash
    elif [ -n "${POSH_VERSION:+x}" ]; then
        :  # pass
    else
        \hash -r
    fi
}

__conda_activate() {
    if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
        # Handle transition from shell activated with conda <= 4.3 to a subsequent activation
        # after conda updated to >= 4.4. See issue #6173.
        PS1="$CONDA_PS1_BACKUP"
        \unset CONDA_PS1_BACKUP
    fi
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$@")" || \return
    \eval "$ask_conda"
    __conda_hashr
}

__conda_reactivate() {
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix reactivate)" || \return
    \eval "$ask_conda"
    __conda_hashr
}

conda() {
    \local cmd="${1-__missing__}"
    case "$cmd" in
        activate|deactivate)
            __conda_activate "$@"
            ;;
        install|update|upgrade|remove|uninstall)
            __conda_exe "$@" || \return
            __conda_reactivate
            ;;
        *)
            __conda_exe "$@"
            ;;
    esac
}

if [ -z "${CONDA_SHLVL+x}" ]; then
    \export CONDA_SHLVL=0
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA:+x}" ] && [ -n "${WINDIR+x}" ]; then
        PATH="$(\dirname "$CONDA_EXE")/condabin${PATH:+":${PATH}"}"
    else
        PATH="$(\dirname "$(\dirname "$CONDA_EXE")")/condabin${PATH:+":${PATH}"}"
    fi
    \export PATH

    # We're not allowing PS1 to be unbound. It must at least be set.
    # However, we're not exporting it, which can cause problems when starting a second shell
    # via a first shell (i.e. starting zsh from bash).
    if [ -z "${PS1+x}" ]; then
        PS1=
    fi
fi
PATH='OBJ/_x86_64_06_avx2:/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'; export PATH; TARG='OBJ/_x86_64_06_avx2'; export TARG; unset 'TARG_BUILD_SCENARIO'; unset 'TARG_COMPILER_FAMILY'; unset 'TARG_EXTRA'; TARG_HOST='farnam'; export TARG_HOST; TARG_MACH='x86_64_06_avx2'; export TARG_MACH; TARG_MACH_DESCRIPT='Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz'; export TARG_MACH_DESCRIPT; unset 'TARG_MPI_FAMILY'; TARG_OS='Linux-3.10.0-1160.42.2.el7.x86_64'; export TARG_OS; TARG_OS_FAMILY='Linux'; export TARG_OS_FAMILY; TARG_SUMMARY='x86_64_06_avx2'; export TARG_SUMMARY; unset 'TARG_TITLE_BAR'; TARG_TITLE_BAR_PAREN=' '; export TARG_TITLE_BAR_PAREN;
conda activate gubbins
PS1='(gubbins)'
export PATH='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'
export CONDA_PREFIX='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_SHLVL='1'
export CONDA_DEFAULT_ENV='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_PROMPT_MODIFIER='(gubbins)'
export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 8 \
    /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln
pgcudahy commented 2 years ago

I added #SBATCH --export=ALL to my hosts.yml config, but still get the original error when submitting from a notebook.

BoPeng commented 2 years ago

The notebook simply runs sbatch to submit jobs so you will need to check if you can manually submit jobs with that module load command.

pgcudahy commented 2 years ago

To try and further debug this I saw that the original error message said Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)

So I added #SBATCH --export=LMOD_SH_DBG_ON=1 to my hosts.yml config, and now it has started working again. I have no idea why this changed anything, but it works. Probably an idiosyncrasy of my cluster and nothing to do with SOS so sorry to take up your time.

Best, Patrick