Closed rubenalv closed 2 months ago
I believe this was fixed by @gcroci2 in a #606, but the patch was not released yet.
I believe this was fixed by @gcroci2 in a #606, but the patch was not released yet.
Exactly, I will release a new patch version before the end of this week. In the meantime, you can pip install the package from the cloned repository, or wait for the new release @rubenalv :)
@DaniBodor @gcroci2 Ohhh, thank you! I'll close the issue as solved.
JFYTK patch 3.0.3 is out :) @rubenalv
@gcroci2, unfortunately I am getting the same error despite the patch! I double-checked and the re-installed deeprank2 version does have the code with dill.dumps / dill.loads. I tried setting nepoch=0 in
trainer.train(
nepoch=epochs,
batch_size=batch_size,
earlystop_patience=earlystop_patience,
earlystop_maxgap=earlystop_maxgap,
min_epoch=min_epoch,
validate=True,
filename=os.path.join(output_path, f"gnn_{task}", "model.pth.tar"),
but to no avail, the OS error appears right away.
Any hints as to how I could go about troubleshooting it in my system, what I should be looking for? Thanks in advance!
@gcroci2, unfortunately I am getting the same error despite the patch!
Weird! To help us investigate and resolve it, then we will need more information from your side.
Thank you, sys specs below:
Operating System: Rocky Linux 8.9 (Green Obsidian) Kernel: Linux 4.18.0-513.24.1.el8_9.x86_64 Architecture: x86-64
Conda environment, running Python 3.10.14. Dependencies below (edited the yml file to allow for CUDA support):
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
abseil-cpp 20211102.0 hd4dd3e8_0
absl-py 2.1.0 py310h06a4308_0
aiohttp 3.9.5 py310h5eee18b_0
aiosignal 1.2.0 pyhd3eb1b0_0
anyio 4.2.0 py310h06a4308_0
argon2-cffi 21.3.0 pyhd3eb1b0_0
argon2-cffi-bindings 21.2.0 py310h7f8727e_0
arrow 1.3.0 pypi_0 pypi
asttokens 2.0.5 pyhd3eb1b0_0
async-lru 2.0.4 py310h06a4308_0
async-timeout 4.0.3 py310h06a4308_0
attrs 23.1.0 py310h06a4308_0
babel 2.11.0 py310h06a4308_0
beautifulsoup4 4.12.2 py310h06a4308_0
biopython 1.83 py310h2372a71_0 conda-forge
blas 1.0 mkl
bleach 4.1.0 pyhd3eb1b0_0
blinker 1.6.2 py310h06a4308_0
blosc 1.21.3 h6a678d5_0
bottleneck 1.3.7 py310ha9d4c09_0
brotli 1.0.9 h5eee18b_8
brotli-bin 1.0.9 h5eee18b_8
brotli-python 1.0.9 py310h6a678d5_8
bump2version 1.0.1 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
c-ares 1.19.1 h5eee18b_0
c-blosc2 2.12.0 h80c7b02_0
ca-certificates 2024.6.2 hbcca054_0 conda-forge
cachetools 5.3.3 py310h06a4308_0
certifi 2024.6.2 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h74dc2b5_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
chart-studio 1.1.0 pyh9f0ad1d_0 conda-forge
click 8.1.7 py310h06a4308_0
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.2.1 py310h06a4308_0
contourpy 1.2.0 py310hdb19cb5_0
coverage 7.5.4 pypi_0 pypi
coveralls 4.0.1 pypi_0 pypi
cryptography 41.0.7 py310hb8475ec_1 conda-forge
cuda-cudart 12.1.105 hd3aeb46_0 conda-forge
cuda-cudart_linux-64 12.1.105 h59595ed_0 conda-forge
cuda-cupti 12.1.105 h59595ed_0 conda-forge
cuda-libraries 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 hd3aeb46_0 conda-forge
cuda-nvtx 12.1.105 h59595ed_0 conda-forge
cuda-opencl 12.1.105 h59595ed_0 conda-forge
cuda-runtime 12.1.0 0 nvidia
cuda-version 12.1 h1f5ad73_3
cycler 0.11.0 pyhd3eb1b0_0
cyrus-sasl 2.1.27 h9033bb2_6 conda-forge
dbus 1.13.18 hb2f20db_0
debugpy 1.6.7 py310h6a678d5_0
decorator 5.1.1 pyhd3eb1b0_0
deeprank2 3.0.4 pypi_0 pypi
defusedxml 0.7.1 pyhd3eb1b0_0
dill 0.3.8 py310h06a4308_0
docopt 0.6.2 pypi_0 pypi
dssp 4.2.2.1 h3fd9d12_0 sbl
exceptiongroup 1.2.0 py310h06a4308_0
executing 0.8.3 pyhd3eb1b0_0
expat 2.6.2 h6a678d5_0
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.13.1 py310h06a4308_0
fontconfig 2.14.1 h4c34cd2_2
fonttools 4.51.0 py310h5eee18b_0
fqdn 1.5.1 pypi_0 pypi
freesasa 2.2.1 py310h2372a71_0 conda-forge
freetype 2.12.1 h4a9f257_0
frozenlist 1.4.0 py310h5eee18b_0
fsspec 2024.3.1 py310h06a4308_0
gettext 0.22.5 h59595ed_2 conda-forge
gettext-tools 0.22.5 h59595ed_2 conda-forge
glib 2.72.1 h6239696_0 conda-forge
glib-tools 2.72.1 h6239696_0 conda-forge
gmp 6.2.1 h295c915_3
gmpy2 2.1.2 py310heeb90bb_0
gnutls 3.6.15 he1e5248_0
google-auth 2.29.0 py310h06a4308_0
google-auth-oauthlib 0.5.2 py310h06a4308_0
grpc-cpp 1.48.2 he1ff14a_1
grpcio 1.48.2 py310he1ff14a_1
gst-plugins-base 1.14.1 h6a678d5_1
gstreamer 1.14.1 h5eee18b_1
h5py 3.11.0 py310hbe37b52_0
hdf5 1.12.1 nompi_h4df4325_104 conda-forge
icu 73.1 h6a678d5_0
idna 3.7 py310h06a4308_0
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
intel-openmp 2021.4.0 h06a4308_3561
ipykernel 6.28.0 py310h06a4308_0
ipython 8.20.0 py310h06a4308_0
isoduration 20.11.0 pypi_0 pypi
jedi 0.18.1 py310h06a4308_1
jinja2 3.1.4 py310h06a4308_0
joblib 1.4.0 py310h06a4308_0
jpeg 9e h5eee18b_1
json5 0.9.6 pyhd3eb1b0_0
jsonpointer 3.0.0 pypi_0 pypi
jsonschema 4.19.2 py310h06a4308_0
jsonschema-specifications 2023.7.1 py310h06a4308_0
jupyter-lsp 2.2.0 py310h06a4308_0
jupyter_client 8.6.0 py310h06a4308_0
jupyter_core 5.5.0 py310h06a4308_0
jupyter_events 0.8.0 py310h06a4308_0
jupyter_server 2.10.0 py310h06a4308_0
jupyter_server_terminals 0.4.4 py310h06a4308_1
jupyterlab 4.0.11 py310h06a4308_0
jupyterlab_pygments 0.1.2 py_0
jupyterlab_server 2.25.1 py310h06a4308_0
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py310h6a678d5_0
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h7b6447c_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libasprintf 0.22.5 h661eb56_2 conda-forge
libasprintf-devel 0.22.5 h661eb56_2 conda-forge
libblas 3.9.0 12_linux64_mkl conda-forge
libbrotlicommon 1.0.9 h5eee18b_8
libbrotlidec 1.0.9 h5eee18b_8
libbrotlienc 1.0.9 h5eee18b_8
libcblas 3.9.0 12_linux64_mkl conda-forge
libcifpp 5.1.0 hf65b397_0 sbl
libclang 14.0.6 default_hc6dbbc7_1
libclang13 14.0.6 default_he11475f_1
libcublas 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufile 1.6.1.9 hd3aeb46_0 conda-forge
libcups 2.3.3 h36d4200_3 conda-forge
libcurand 10.3.2.106 hd3aeb46_0 conda-forge
libcurl 8.1.2 h409715c_0 conda-forge
libcusolver 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libdeflate 1.17 h5eee18b_1
libedit 3.1.20230828 h5eee18b_0
libev 4.33 h7f8727e_1
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h77fa898_7 conda-forge
libgettextpo 0.22.5 h59595ed_2 conda-forge
libgettextpo-devel 0.22.5 h59595ed_2 conda-forge
libgfortran-ng 13.2.0 h69a702a_7 conda-forge
libgfortran5 13.2.0 hca663fb_7 conda-forge
libglib 2.72.1 h2d90d5f_0 conda-forge
libgomp 13.2.0 h77fa898_7 conda-forge
libiconv 1.16 h5eee18b_3
libidn2 2.3.4 h5eee18b_0
libjpeg-turbo 2.0.0 h9bf148f_0 pytorch
liblapack 3.9.0 12_linux64_mkl conda-forge
libllvm14 14.0.6 hcd5def8_4 conda-forge
libmcfp 1.2.3 h3fd9d12_0 sbl
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnpp 12.0.2.50 0 nvidia
libntlm 1.4 h7f98852_1002 conda-forge
libnvjitlink 12.1.105 hd3aeb46_0 conda-forge
libnvjpeg 12.1.1.14 0 nvidia
libpng 1.6.39 h5eee18b_0
libpq 12.17 hdbd6064_0
libprotobuf 3.20.3 he621ea3_0
libsodium 1.0.18 h7b6447c_0
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.2.0 hc0a3c3a_7 conda-forge
libtasn1 4.19.0 h5eee18b_0
libtiff 4.5.1 h6a678d5_0
libunistring 0.9.10 h27cfd23_0
libuuid 1.41.5 h5eee18b_0
libwebp-base 1.3.2 h5eee18b_0
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 h5eee18b_1
libxml2 2.10.4 hfdd30dd_2
libzlib 1.2.13 h4ab18f5_6 conda-forge
llvm-openmp 14.0.6 h9e868ea_0
lz4-c 1.9.4 h6a678d5_1
lzo 2.10 h7b6447c_2
markdown 3.4.1 py310h06a4308_0
markov_clustering 0.0.6 py_0 bioconda
markupsafe 2.1.3 py310h5eee18b_0
matplotlib 3.8.4 py310h06a4308_0
matplotlib-base 3.8.4 py310h1128e8f_0
matplotlib-inline 0.1.6 py310h06a4308_0
mistune 2.0.4 py310h06a4308_0
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py310h7f8727e_0
mkl_fft 1.3.1 py310hd6ae3a3_0
mkl_random 1.2.2 py310h00e6091_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpmath 1.3.0 py310h06a4308_0
msms 2.6.1 h9ee0642_3 bioconda
multidict 6.0.4 py310h5eee18b_0
mysql 5.7.20 hf484d3e_1001 conda-forge
nbclient 0.8.0 py310h06a4308_0
nbconvert 7.10.0 py310h06a4308_0
nbformat 5.9.2 py310h06a4308_0
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 py310h06a4308_0
nettle 3.7.3 hbbd107a_1
networkx 3.2.1 py310h06a4308_0
notebook 7.0.8 py310h06a4308_0
notebook-shim 0.2.3 py310h06a4308_0
numexpr 2.8.4 py310h8879344_0
numpy 1.26.4 pypi_0 pypi
numpy-base 1.24.3 py310h8e6c178_0
oauthlib 3.2.2 py310h06a4308_0
ocl-icd 2.3.2 h5eee18b_1
openh264 2.1.1 h4ff587b_0
openjpeg 2.4.0 h3ad879b_0
openssl 3.3.1 h4ab18f5_0 conda-forge
overrides 7.4.0 py310h06a4308_0
packaging 23.2 py310h06a4308_0
pandas 2.2.2 py310hf9f9076_1 conda-forge
pandocfilters 1.5.0 pyhd3eb1b0_0
parso 0.8.3 pyhd3eb1b0_0
pcre 8.45 h295c915_0
pdb2sql 0.5.3 pyhd8ed1ab_0 conda-forge
pexpect 4.8.0 pyhd3eb1b0_3
pillow 10.3.0 py310h5eee18b_0
pip 24.0 py310h06a4308_0
platformdirs 3.10.0 py310h06a4308_0
plotly 5.19.0 py310h2f386ee_0
pluggy 1.5.0 pyhd8ed1ab_0 conda-forge
ply 3.11 py310h06a4308_0
pooch 1.8.2 pyhd8ed1ab_0 conda-forge
prometheus_client 0.14.1 py310h06a4308_0
prompt-toolkit 3.0.43 py310h06a4308_0
prompt_toolkit 3.0.43 hd3eb1b0_0
protobuf 3.20.3 py310h6a678d5_0
psutil 5.9.0 py310h5eee18b_0
ptyprocess 0.7.0 pyhd3eb1b0_2
pure_eval 0.2.2 pyhd3eb1b0_0
py-cpuinfo 9.0.0 py310h06a4308_0
pyarrow 16.1.0 pypi_0 pypi
pyasn1 0.4.8 pyhd3eb1b0_0
pyasn1-modules 0.2.8 py_0
pycodestyle 2.12.0 pypi_0 pypi
pycparser 2.21 pyhd3eb1b0_0
pyg 2.5.2 py310_torch_2.1.0_cu121 pyg
pygments 2.15.1 py310h06a4308_1
pyjwt 2.8.0 py310h06a4308_0
pyopenssl 23.2.0 py310h06a4308_0
pyparsing 3.0.9 py310h06a4308_0
pyqt 5.15.10 py310h6a678d5_0
pyqt5-sip 12.13.0 py310h5eee18b_0
pysocks 1.7.1 py310h06a4308_0
pytest 8.2.2 pyhd8ed1ab_0 conda-forge
pytest-cov 5.0.0 pypi_0 pypi
pytest-runner 6.0.1 pypi_0 pypi
python 3.10.14 h955ad1f_1
python-dateutil 2.9.0post0 py310h06a4308_2
python-fastjsonschema 2.16.2 py310h06a4308_0
python-json-logger 2.0.7 py310h06a4308_0
python-louvain 0.16 pyhd8ed1ab_0 conda-forge
python-tzdata 2023.3 pyhd3eb1b0_0
python_abi 3.10 2_cp310 conda-forge
pytorch 2.1.1 py3.10_cuda12.1_cudnn8.9.2_0 pytorch
pytorch-cluster 1.6.3 py310_torch_2.1.0_cu121 pyg
pytorch-cuda 12.1 ha16c6d3_5 pytorch
pytorch-mutex 1.0 cuda pytorch
pytorch-scatter 2.1.2 py310_torch_2.1.0_cu121 pyg
pytorch-sparse 0.6.18 py310_torch_2.1.0_cu121 pyg
pytorch-spline-conv 1.2.2 py310_torch_2.1.0_cu121 pyg
pytz 2024.1 py310h06a4308_0
pyyaml 6.0.1 py310h5eee18b_0
pyzmq 25.1.2 py310h6a678d5_0
qt-main 5.15.2 h53bd1ea_10
re2 2022.04.01 h295c915_0
readline 8.2 h5eee18b_0
referencing 0.30.2 py310h06a4308_0
requests 2.32.2 py310h06a4308_0
requests-oauthlib 1.3.0 py_0
retrying 1.3.3 pyhd3eb1b0_2
rfc3339-validator 0.1.4 py310h06a4308_0
rfc3986-validator 0.1.1 py310h06a4308_0
rpds-py 0.10.6 py310hb02cf49_0
rsa 4.7.2 pyhd3eb1b0_1
ruff 0.3.5 py310hab49468_0
scikit-learn 1.4.2 py310h1128e8f_1
scipy 1.13.1 py310h93e2701_0 conda-forge
send2trash 1.8.2 py310h06a4308_0
setuptools 69.5.1 py310h06a4308_0
sip 6.7.12 py310h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
sniffio 1.3.0 py310h06a4308_0
soupsieve 2.5 py310h06a4308_0
sqlite 3.45.3 h5eee18b_0
stack_data 0.2.0 pyhd3eb1b0_0
sympy 1.12 py310h06a4308_0
tables 3.9.2 py310h0016290_0
tenacity 8.2.2 py310h06a4308_0
tensorboard 2.12.1 py310h06a4308_0
tensorboard-data-server 0.7.0 py310h52d8a92_0
tensorboard-plugin-wit 1.8.1 py310h06a4308_0
terminado 0.17.1 py310h06a4308_0
threadpoolctl 2.2.0 pyh0d69192_0
tinycss2 1.2.1 py310h06a4308_0
tk 8.6.14 h39e8969_0
tomli 2.0.1 py310h06a4308_0
torchaudio 2.1.1 py310_cu121 pytorch
torchtriton 2.1.0 py310 pytorch
torchvision 0.16.1 py310_cu121 pytorch
tornado 6.3.3 py310h5eee18b_0
tqdm 4.66.4 py310h2f386ee_0
traitlets 5.7.1 py310h06a4308_0
types-python-dateutil 2.9.0.20240316 pypi_0 pypi
typing-extensions 4.11.0 py310h06a4308_0
typing_extensions 4.11.0 py310h06a4308_0
tzdata 2024a h04d1e81_0
unicodedata2 15.1.0 py310h5eee18b_0
uri-template 1.3.0 pypi_0 pypi
urllib3 2.2.1 py310h06a4308_0
wcwidth 0.2.5 pyhd3eb1b0_0
webcolors 24.6.0 pypi_0 pypi
webencodings 0.5.1 py310h06a4308_1
websocket-client 1.8.0 py310h06a4308_0
werkzeug 3.0.3 py310h06a4308_0
wheel 0.43.0 py310h06a4308_0
xz 5.4.6 h5eee18b_1
yaml 0.2.5 h7b6447c_0
yarl 1.9.3 py310h5eee18b_0
zeromq 4.3.5 h6a678d5_0
zlib 1.2.13 h4ab18f5_6 conda-forge
zlib-ng 2.0.7 h5eee18b_0
zstd 1.5.5 hc292b87_2
We have a privileged user for installations, and regular users. I noticed that writing privileges in deeprank2/ are required to at least run pytest
(eg write test.pth.tar
). The OS error seems independent of privileges though.
I wonder if for some reason the object features_transform
in features_transform_to_save = copy.deepcopy(self.features_transform)
has a problem, so there is no source code to actually extract, so I am trying to figure out how to check it.
@gcroci2, I found the reason for the problem: I was running the code (https://github.com/DeepRank/deeprank2/blob/main/tutorials/training.ipynb) from the Python command line (not from the original Jupiter notebook). I copied it to a script and run python script.py
, and the training completed as expected.
I don't know enough Python to understand why, because I could use inspect.getsource() from command line in a mock serialized/deserialized object. If you knew why and could point me to an explanation it would be great!
Perhaps it would be worth it to add a note to not run in command line in the tutorial? I think the issue is OK to be closed now.
@gcroci2, I found the reason for the problem: I was running the code (https://github.com/DeepRank/deeprank2/blob/main/tutorials/training.ipynb) from the Python command line (not from the original Jupiter notebook). I copied it to a script and run
python script.py
, and the training completed as expected.I don't know enough Python to understand why, because I could use inspect.getsource() from command line in a mock serialized/deserialized object. If you knew why and could point me to an explanation it would be great!
@rubenalv It sounds like the issue is related to how inspect.getsource()
interacts with code executed in different environments.
If I understand correctly, the code was trowing an error about inspect.getsource()
when you run the code from the Python command line. However, it worked when running the same code within a Jupyter notebook or a Python script.
inspect.getsource()
work more reliably within a notebook. The notebook environment can store the cell's code in a way that inspect can access it, even if it's not coming from a traditional script file.inspect.getsource()
can locate the source code of functions and classes because it knows the exact file and location. If the function or class definition is imported from another module, inspect.getsource()
can still locate it, assuming the source file is accessible.So to conclude, my guess is that when you run the code from the Python command line you are defining an "interactive" job on the cluster without a source file, and inspect.getsource()
fails because it can't locate the source code.
Perhaps it would be worth it to add a note to not run in command line in the tutorial? I think the issue is OK to be closed now.
Good suggestion, thanks :) I opened a PR (#620) for it.
All much appreciated, thank you for your work to make DeepRank2 accesible!
I downloaded the pre_processed data and tried running the tutorial here https://github.com/DeepRank/deeprank2/blob/main/tutorials/training.ipynb
At the stage of
The output is:
I have deeprank2 installed in a cluster, with permissions rwxr.x... and I run it with my user that belongs to the group (ie running it with permissions r.x). I tried running it with full rwx as well, but got the same error.
Also, I noticed that the file in https://github.com/DeepRank/deeprank2/blob/main/deeprank2/trainer.py is not identical to the file that gets installed. At least, in the save_model() function they differ: In the file:
But in github:
Any hints as to the OS error?