Closed hscannell closed 2 years ago
There is a typo here
Could that be relevant?
Nice catch! That line was hastily added while I was trying to debug. It should be removed.
Otherwise, the problem still stands.
Here are my ~/.bashrc
file contents:
# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/burg/opt/anaconda3-2020.11/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/burg/opt/anaconda3-2020.11/anaconda3/etc/profile.d/conda.sh" ]; then
. "/burg/opt/anaconda3-2020.11/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/burg/opt/anaconda3-2020.11/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
My best guess would be that the environment is not actually loading correctly.
Add the following lines to your script before calling the python script to debug:
which python
conda info
conda list
**For reference
/burg/opt/anaconda3-2021.05/bin/python
2.
active environment : lcs-ml
active env location : /burg/home/hs3277/.conda/envs/lcs-ml
shell level : 2
user config file : /burg/home/hs3277/.condarc
populated config files :
conda version : 4.9.2
conda-build version : 3.20.5
python version : 3.8.5.final.0
virtual packages : __glibc=2.28=0
__unix=0=0
__archspec=1=x86_64
base environment : /burg/opt/anaconda3-2020.11/anaconda3 (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /burg/opt/anaconda3-2020.11/anaconda3/pkgs
/burg/home/hs3277/.conda/pkgs
envs directories : /burg/home/hs3277/.conda/envs
/burg/opt/anaconda3-2020.11/anaconda3/envs
platform : linux-64
user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Linux/4.18.0-193.el8.x86_64 rhel/8.2 glibc/2.28
UID:GID : 545216:500
netrc file : None
offline mode : False
3.
# packages in environment at /burg/home/hs3277/.conda/envs/lcs-ml:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
argcomplete 1.12.3 pyhd3eb1b0_0
argon2-cffi 20.1.0 py37h27cfd23_1
asciitree 0.3.3 py_2
asn1crypto 1.4.0 py_0
async_generator 1.10 py37h28b3542_0
atk 2.36.0 h06a4308_0
atk-1.0 2.36.0 h28cd5cc_0
attrs 21.2.0 pyhd3eb1b0_0
backcall 0.2.0 pyhd3eb1b0_0
blas 1.0 openblas
bleach 4.0.0 pyhd3eb1b0_0
bokeh 2.3.2 py37h06a4308_0
brotli 1.0.9 he6710b0_2
brotlipy 0.7.0 py37h27cfd23_1003
bzip2 1.0.8 h7b6447c_0
ca-certificates 2021.10.8 ha878542_0 conda-forge
cairo 1.16.0 h18b612c_1001 conda-forge
certifi 2021.10.8 py37h89c1867_1 conda-forge
cffi 1.14.0 py37h2e261b9_0
cftime 1.5.0 py37h6323ea4_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.0.3 pyhd3eb1b0_0
cloudpickle 2.0.0 pyhd3eb1b0_0
colorcet 2.0.6 pyhd3eb1b0_0
cryptography 2.3.1 py37hc365091_0
curl 7.64.0 h646f8bb_0 conda-forge
cycler 0.10.0 py37_0
cython 0.29.24 py37hcd2ae1e_0 conda-forge
cytoolz 0.11.0 py37h7b6447c_0
dask 2021.10.0 pyhd8ed1ab_0 conda-forge
dask-core 2021.10.0 pyhd3eb1b0_0
datashader 0.13.0 pyhd3eb1b0_1
datashape 0.5.4 py37h06a4308_1
dbus 1.13.18 hb2f20db_0
debugpy 1.4.1 py37h295c915_0
decorator 5.1.0 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
distributed 2021.10.0 py37h06a4308_0
entrypoints 0.3 py37_0
expat 2.4.1 h2531618_2
fasteners 0.16.3 pyhd3eb1b0_0
fontconfig 2.13.1 he4413a7_1000 conda-forge
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.11.0 h70c0345_0
fribidi 1.0.10 h7b6447c_0
fsspec 2021.8.1 pyhd3eb1b0_0
gdk-pixbuf 2.42.2 h19a9c64_1 conda-forge
gettext 0.21.0 hf68c758_0
giflib 5.2.1 h7b6447c_0
glib 2.66.3 h58526e2_0 conda-forge
gobject-introspection 1.66.1 py37h9019ea4_3 conda-forge
graphite2 1.3.14 h23475e2_0
gstreamer 1.14.5 h36ae1b5_2 conda-forge
gtk2 2.24.32 h90f3771_0 conda-forge
h5netcdf 0.11.0 pyhd8ed1ab_0 conda-forge
h5py 2.10.0 py37h7918eee_0
harfbuzz 2.4.0 h37c48d4_1 conda-forge
hdf4 4.2.13 h3ca952b_2
hdf5 1.10.4 hb1b8bf9_0
heapdict 1.0.1 pyhd3eb1b0_0
holoviews 1.14.6 pyhd3eb1b0_1
icu 58.2 he6710b0_3
idna 3.2 pyhd3eb1b0_0
importlib-metadata 4.8.1 py37h06a4308_0
importlib_metadata 4.8.1 hd3eb1b0_0
ipykernel 6.4.2 py37h6531663_0 conda-forge
ipython 7.29.0 py37hb070fc8_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
jedi 0.18.0 py37h06a4308_1
jinja2 3.0.2 pyhd3eb1b0_0
jpeg 9d h7f8727e_0
jsonschema 3.2.0 pyhd3eb1b0_2
jupyter_client 7.0.1 pyhd3eb1b0_0
jupyter_core 4.8.1 py37h06a4308_0
jupyterlab_pygments 0.1.2 py_0
kiwisolver 1.3.1 py37h2531618_0
krb5 1.16.3 hc83ff2d_1000 conda-forge
lcms2 2.12 h3be6417_0
libblas 3.9.0 8_openblas conda-forge
libcblas 3.9.0 8_openblas conda-forge
libcurl 7.64.0 h01ee5af_0 conda-forge
libedit 3.1.20210216 h27cfd23_1
libffi 3.2.1 hf484d3e_1007
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libglib 2.66.3 hbe7bbb4_0 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 8_openblas conda-forge
libllvm10 10.0.1 hbcb73fb_5
libnetcdf 4.6.2 hbdf4f91_1001 conda-forge
libopenblas 0.3.12 pthreads_hb3c22a3_1 conda-forge
libpng 1.6.37 hbc83047_0
libsodium 1.0.18 h7b6447c_0
libssh2 1.8.0 h1ad7b7a_1003 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17
libtiff 4.2.0 h85742a9_0
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp 1.2.0 h89dd481_0
libwebp-base 1.2.0 h27cfd23_0
libxcb 1.14 h7b6447c_0
libxml2 2.9.12 h03d6c58_0
llvmlite 0.36.0 py37h612dafd_4
locket 0.2.1 py37h06a4308_1
lz4-c 1.9.3 h295c915_1
markdown 3.3.4 py37h06a4308_0
markupsafe 2.0.1 py37h27cfd23_0
matplotlib 3.4.3 py37h89c1867_1 conda-forge
matplotlib-base 3.4.3 py37hbbc1b5f_0
matplotlib-inline 0.1.2 pyhd3eb1b0_2
mistune 0.8.4 py37h14c3975_1001
msgpack-python 1.0.2 py37hff7bd54_1
multipledispatch 0.6.0 py37_0
munkres 1.1.4 py_0
nbclient 0.5.3 pyhd3eb1b0_0
nbconvert 6.1.0 py37h06a4308_0
nbformat 5.1.3 pyhd3eb1b0_0
ncurses 6.2 he6710b0_1
nest-asyncio 1.5.1 pyhd3eb1b0_0
netcdf4 1.5.1.2 py37had58050_0 conda-forge
ninja 1.10.2 hff7bd54_1
notebook 6.4.5 py37h06a4308_0
numba 0.53.1 py37ha9443f7_0
numcodecs 0.8.0 py37h2531618_0
numpy 1.21.3 py37h31617e3_0 conda-forge
olefile 0.46 py37_0
openssl 1.0.2u h516909a_0 conda-forge
packaging 21.0 pyhd3eb1b0_0
pandas 1.2.0 py37ha9443f7_0
pandocfilters 1.4.3 py37h06a4308_1
panel 0.8.3 pypi_0 pypi
pango 1.42.4 h7062337_4 conda-forge
param 1.11.1 pyhd3eb1b0_0
parso 0.8.2 pyhd3eb1b0_0
partd 1.2.0 pyhd3eb1b0_0
pcre 8.45 h295c915_0
pexpect 4.8.0 pyhd3eb1b0_3
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 8.4.0 py37h5aabda8_0
pip 21.0.1 py37h06a4308_0
pixman 0.38.0 h516909a_1003 conda-forge
prometheus_client 0.11.0 pyhd3eb1b0_0
prompt-toolkit 3.0.20 pyhd3eb1b0_0
psutil 5.8.0 py37h27cfd23_1
ptyprocess 0.7.0 pyhd3eb1b0_2
pycparser 2.20 py_2
pyct 0.4.8 py37_0
pyfftw 0.11.1 py37h240ffea_1002 conda-forge
pygments 2.10.0 pyhd3eb1b0_0
pyopenssl 19.0.0 py37_0
pyparsing 3.0.4 pyhd3eb1b0_0
pyqg 0.4.0 py37hb1e94ed_0 conda-forge
pyqt 5.6.0 py37h13b7fb3_1008 conda-forge
pyrsistent 0.18.0 py37heee7806_0
pysocks 1.7.1 py37_1
python 3.7.0 hd21baee_1006 conda-forge
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.7 2_cp37m conda-forge
pytz 2021.3 pyhd3eb1b0_0
pyviz_comms 2.0.2 pyhd3eb1b0_0
pyyaml 6.0 py37h5e8e339_0 conda-forge
pyzmq 22.2.1 py37h295c915_1
qt 5.6.2 hce4f676_1013 conda-forge
readline 7.0 h7b6447c_5
requests 2.26.0 pyhd3eb1b0_0
scipy 1.3.0 py37he2b7bc3_0
send2trash 1.8.0 pyhd3eb1b0_1
setuptools 58.0.4 py37h06a4308_0
sip 4.18.1 py37hf484d3e_1000 conda-forge
six 1.16.0 pyhd3eb1b0_0
sortedcontainers 2.4.0 pyhd3eb1b0_0
sqlite 3.28.0 h8b20d00_0 conda-forge
tbb 2020.3 hfd86e86_0
tblib 1.7.0 pyhd3eb1b0_0
terminado 0.9.4 py37h06a4308_0
testpath 0.5.0 pyhd3eb1b0_0
tk 8.6.11 h1ccaba5_0
toolz 0.11.1 pyhd3eb1b0_0
tornado 6.1 py37h27cfd23_0
tqdm 4.62.3 pyhd3eb1b0_1
traitlets 5.1.0 pyhd3eb1b0_0
typing_extensions 3.10.0.2 pyh06a4308_0
urllib3 1.26.7 pyhd3eb1b0_0
wcwidth 0.2.5 pyhd3eb1b0_0
webencodings 0.5.1 py37_1
wheel 0.37.0 pyhd3eb1b0_1
xarray 0.19.0 pyhd8ed1ab_1 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-libxt 1.2.1 h7f98852_2 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h27cfd23_1007
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zarr 2.8.1 pyhd3eb1b0_0
zeromq 4.3.4 h2531618_0
zict 2.0.0 pyhd3eb1b0_0
zipp 3.6.0 pyhd3eb1b0_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.9 haebb681_0
/burg/opt/anaconda3-2021.05/bin/python
So this is not the python from your conda environment. That python should be in /burg/home/hs3277/.conda/envs/lcs-ml
. This explains why it is not working.
I discovered that the anaconda version was not the same in bashrc vs. what I was loading in the script.
I changed this in the bash script and things seem to work!
module load anaconda/3-2021.05
conda init bash
I am getting the following error when running either spin_up.sh or ensemble_generator.sh in batch mode.
To run either process, I type the following at the terminal:
sbatch spin_up.sh
sbatch --array=1-3 ensemble_generator.sh
My environment is
lcs-ml
and containspyqg
version 0.4.0, and I specify the exact environment in my batch script, so I'm at a loss as to why this error is occurring.Using the
lcs-ml
in the notebook version of the script causes no errors.