Bug: OSError: could not get source code

rubenalv commented 2 months ago

I downloaded the pre_processed data and tried running the tutorial here https://github.com/DeepRank/deeprank2/blob/main/tutorials/training.ipynb

At the stage of

trainer.train(
    nepoch=epochs,
    batch_size=batch_size,
    earlystop_patience=earlystop_patience,
    earlystop_maxgap=earlystop_maxgap,
    min_epoch=min_epoch,
    validate=True,
    filename=os.path.join(output_path, f"gnn_{task}", "model.pth.tar"),
)

The output is:

INFO:deeprank2.trainer:Training set loaded
INFO:deeprank2.trainer:Validation set loaded
INFO:deeprank2.trainer:Epoch 0:
INFO:deeprank2.trainer:training loss 0.8269949356714884 | time 0.3814964294433594
INFO:deeprank2.trainer:validation loss 0.8072875671916537 | time 0.0742034912109375
INFO:deeprank2.trainer:Epoch 1:
INFO:deeprank2.trainer:training loss 0.7365012235111661 | time 0.36728739738464355
INFO:deeprank2.trainer:validation loss 0.6814224786228604 | time 0.07352352142333984
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/apps/software/Anaconda3/2022.05/envs/deeprank2/lib/python3.10/site-packages/deeprank2/trainer.py", line 649, in train
    checkpoint_model = self._save_model()
  File "/apps/software/Anaconda3/2022.05/envs/deeprank2/lib/python3.10/site-packages/deeprank2/trainer.py", line 949, in _save_model
    str_expr = inspect.getsource(key["transform"])
  File "/apps/software/Anaconda3/2022.05/envs/deeprank2/lib/python3.10/inspect.py", line 1139, in getsource
    lines, lnum = getsourcelines(object)
  File "/apps/software/Anaconda3/2022.05/envs/deeprank2/lib/python3.10/inspect.py", line 1121, in getsourcelines
    lines, lnum = findsource(object)
  File "/apps/software/Anaconda3/2022.05/envs/deeprank2/lib/python3.10/inspect.py", line 958, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

I have deeprank2 installed in a cluster, with permissions rwxr.x... and I run it with my user that belongs to the group (ie running it with permissions r.x). I tried running it with full rwx as well, but got the same error.

Also, I noticed that the file in https://github.com/DeepRank/deeprank2/blob/main/deeprank2/trainer.py is not identical to the file that gets installed. At least, in the save_model() function they differ: In the file:

     if features_transform_to_save:
            for key in features_transform_to_save.values():
                if key["transform"] is None:
                    continue
                str_expr = inspect.getsource(key["transform"])
                match = re.search(r"[\"|\']transform[\"|\']:.*(lambda.*).*,.*[\"|\']standardize[\"|\'].*", str_expr).group(1)
                key["transform"] = match

But in github:

        if features_transform_to_save:
            for key in features_transform_to_save.values():
                if key["transform"] is None:
                    continue
                # Serialize the function
                serialized_func = dill.dumps(key["transform"])
                # Deserialize the function
                deserialized_func = dill.loads(serialized_func)  # noqa: S301
                str_expr = inspect.getsource(deserialized_func)
                match = re.search(r"[\"|\']transform[\"|\']:.*(lambda.*).*,.*[\"|\']standardize[\"|\'].*", str_expr).group(1)
                key["transform"] = match

Any hints as to the OS error?

DaniBodor commented 2 months ago

I believe this was fixed by @gcroci2 in a #606, but the patch was not released yet.

gcroci2 commented 2 months ago

I believe this was fixed by @gcroci2 in a #606, but the patch was not released yet.

Exactly, I will release a new patch version before the end of this week. In the meantime, you can pip install the package from the cloned repository, or wait for the new release @rubenalv :)

rubenalv commented 2 months ago

@DaniBodor @gcroci2 Ohhh, thank you! I'll close the issue as solved.

gcroci2 commented 2 months ago

JFYTK patch 3.0.3 is out :) @rubenalv

rubenalv commented 2 months ago

@gcroci2, unfortunately I am getting the same error despite the patch! I double-checked and the re-installed deeprank2 version does have the code with dill.dumps / dill.loads. I tried setting nepoch=0 in

trainer.train(
    nepoch=epochs,
    batch_size=batch_size,
    earlystop_patience=earlystop_patience,
    earlystop_maxgap=earlystop_maxgap,
    min_epoch=min_epoch,
    validate=True,
    filename=os.path.join(output_path, f"gnn_{task}", "model.pth.tar"),

but to no avail, the OS error appears right away.

Any hints as to how I could go about troubleshooting it in my system, what I should be looking for? Thanks in advance!

gcroci2 commented 2 months ago

@gcroci2, unfortunately I am getting the same error despite the patch!

Weird! To help us investigate and resolve it, then we will need more information from your side.

Which operating system are you using (e.g., Windows 10, macOS Big Sur, Ubuntu 20.04)?
Please specify the version of Python you have installed (python --version).
Are you using a virtual environment (e.g., Conda, venv), or are you working in a global Python environment?
Could you provide a list of dependencies installed in your environment? You can generate this list by running pip freeze or conda list depending on your environment.
Any other information about your setup or environment that you think might be relevant.

rubenalv commented 2 months ago

Thank you, sys specs below:

Operating System: Rocky Linux 8.9 (Green Obsidian) Kernel: Linux 4.18.0-513.24.1.el8_9.x86_64 Architecture: x86-64

Conda environment, running Python 3.10.14. Dependencies below (edited the yml file to allow for CUDA support):

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
abseil-cpp                20211102.0           hd4dd3e8_0
absl-py                   2.1.0           py310h06a4308_0
aiohttp                   3.9.5           py310h5eee18b_0
aiosignal                 1.2.0              pyhd3eb1b0_0
anyio                     4.2.0           py310h06a4308_0
argon2-cffi               21.3.0             pyhd3eb1b0_0
argon2-cffi-bindings      21.2.0          py310h7f8727e_0
arrow                     1.3.0                    pypi_0    pypi
asttokens                 2.0.5              pyhd3eb1b0_0
async-lru                 2.0.4           py310h06a4308_0
async-timeout             4.0.3           py310h06a4308_0
attrs                     23.1.0          py310h06a4308_0
babel                     2.11.0          py310h06a4308_0
beautifulsoup4            4.12.2          py310h06a4308_0
biopython                 1.83            py310h2372a71_0    conda-forge
blas                      1.0                         mkl
bleach                    4.1.0              pyhd3eb1b0_0
blinker                   1.6.2           py310h06a4308_0
blosc                     1.21.3               h6a678d5_0
bottleneck                1.3.7           py310ha9d4c09_0
brotli                    1.0.9                h5eee18b_8
brotli-bin                1.0.9                h5eee18b_8
brotli-python             1.0.9           py310h6a678d5_8
bump2version              1.0.1                    pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6
c-ares                    1.19.1               h5eee18b_0
c-blosc2                  2.12.0               h80c7b02_0
ca-certificates           2024.6.2             hbcca054_0    conda-forge
cachetools                5.3.3           py310h06a4308_0
certifi                   2024.6.2           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h74dc2b5_0
charset-normalizer        2.0.4              pyhd3eb1b0_0
chart-studio              1.1.0              pyh9f0ad1d_0    conda-forge
click                     8.1.7           py310h06a4308_0
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.2.1           py310h06a4308_0
contourpy                 1.2.0           py310hdb19cb5_0
coverage                  7.5.4                    pypi_0    pypi
coveralls                 4.0.1                    pypi_0    pypi
cryptography              41.0.7          py310hb8475ec_1    conda-forge
cuda-cudart               12.1.105             hd3aeb46_0    conda-forge
cuda-cudart_linux-64      12.1.105             h59595ed_0    conda-forge
cuda-cupti                12.1.105             h59595ed_0    conda-forge
cuda-libraries            12.1.0                        0    nvidia
cuda-nvrtc                12.1.105             hd3aeb46_0    conda-forge
cuda-nvtx                 12.1.105             h59595ed_0    conda-forge
cuda-opencl               12.1.105             h59595ed_0    conda-forge
cuda-runtime              12.1.0                        0    nvidia
cuda-version              12.1                 h1f5ad73_3
cycler                    0.11.0             pyhd3eb1b0_0
cyrus-sasl                2.1.27               h9033bb2_6    conda-forge
dbus                      1.13.18              hb2f20db_0
debugpy                   1.6.7           py310h6a678d5_0
decorator                 5.1.1              pyhd3eb1b0_0
deeprank2                 3.0.4                    pypi_0    pypi
defusedxml                0.7.1              pyhd3eb1b0_0
dill                      0.3.8           py310h06a4308_0
docopt                    0.6.2                    pypi_0    pypi
dssp                      4.2.2.1              h3fd9d12_0    sbl
exceptiongroup            1.2.0           py310h06a4308_0
executing                 0.8.3              pyhd3eb1b0_0
expat                     2.6.2                h6a678d5_0
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.13.1          py310h06a4308_0
fontconfig                2.14.1               h4c34cd2_2
fonttools                 4.51.0          py310h5eee18b_0
fqdn                      1.5.1                    pypi_0    pypi
freesasa                  2.2.1           py310h2372a71_0    conda-forge
freetype                  2.12.1               h4a9f257_0
frozenlist                1.4.0           py310h5eee18b_0
fsspec                    2024.3.1        py310h06a4308_0
gettext                   0.22.5               h59595ed_2    conda-forge
gettext-tools             0.22.5               h59595ed_2    conda-forge
glib                      2.72.1               h6239696_0    conda-forge
glib-tools                2.72.1               h6239696_0    conda-forge
gmp                       6.2.1                h295c915_3
gmpy2                     2.1.2           py310heeb90bb_0
gnutls                    3.6.15               he1e5248_0
google-auth               2.29.0          py310h06a4308_0
google-auth-oauthlib      0.5.2           py310h06a4308_0
grpc-cpp                  1.48.2               he1ff14a_1
grpcio                    1.48.2          py310he1ff14a_1
gst-plugins-base          1.14.1               h6a678d5_1
gstreamer                 1.14.1               h5eee18b_1
h5py                      3.11.0          py310hbe37b52_0
hdf5                      1.12.1          nompi_h4df4325_104    conda-forge
icu                       73.1                 h6a678d5_0
idna                      3.7             py310h06a4308_0
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
intel-openmp              2021.4.0          h06a4308_3561
ipykernel                 6.28.0          py310h06a4308_0
ipython                   8.20.0          py310h06a4308_0
isoduration               20.11.0                  pypi_0    pypi
jedi                      0.18.1          py310h06a4308_1
jinja2                    3.1.4           py310h06a4308_0
joblib                    1.4.0           py310h06a4308_0
jpeg                      9e                   h5eee18b_1
json5                     0.9.6              pyhd3eb1b0_0
jsonpointer               3.0.0                    pypi_0    pypi
jsonschema                4.19.2          py310h06a4308_0
jsonschema-specifications 2023.7.1        py310h06a4308_0
jupyter-lsp               2.2.0           py310h06a4308_0
jupyter_client            8.6.0           py310h06a4308_0
jupyter_core              5.5.0           py310h06a4308_0
jupyter_events            0.8.0           py310h06a4308_0
jupyter_server            2.10.0          py310h06a4308_0
jupyter_server_terminals  0.4.4           py310h06a4308_1
jupyterlab                4.0.11          py310h06a4308_0
jupyterlab_pygments       0.1.2                      py_0
jupyterlab_server         2.25.1          py310h06a4308_0
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310h6a678d5_0
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100                h7b6447c_0
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
lerc                      3.0                  h295c915_0
libasprintf               0.22.5               h661eb56_2    conda-forge
libasprintf-devel         0.22.5               h661eb56_2    conda-forge
libblas                   3.9.0            12_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h5eee18b_8
libbrotlidec              1.0.9                h5eee18b_8
libbrotlienc              1.0.9                h5eee18b_8
libcblas                  3.9.0            12_linux64_mkl    conda-forge
libcifpp                  5.1.0                hf65b397_0    sbl
libclang                  14.0.6          default_hc6dbbc7_1
libclang13                14.0.6          default_he11475f_1
libcublas                 12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufile                 1.6.1.9              hd3aeb46_0    conda-forge
libcups                   2.3.3                h36d4200_3    conda-forge
libcurand                 10.3.2.106           hd3aeb46_0    conda-forge
libcurl                   8.1.2                h409715c_0    conda-forge
libcusolver               11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libdeflate                1.17                 h5eee18b_1
libedit                   3.1.20230828         h5eee18b_0
libev                     4.33                 h7f8727e_1
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h77fa898_7    conda-forge
libgettextpo              0.22.5               h59595ed_2    conda-forge
libgettextpo-devel        0.22.5               h59595ed_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_7    conda-forge
libgfortran5              13.2.0               hca663fb_7    conda-forge
libglib                   2.72.1               h2d90d5f_0    conda-forge
libgomp                   13.2.0               h77fa898_7    conda-forge
libiconv                  1.16                 h5eee18b_3
libidn2                   2.3.4                h5eee18b_0
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
liblapack                 3.9.0            12_linux64_mkl    conda-forge
libllvm14                 14.0.6               hcd5def8_4    conda-forge
libmcfp                   1.2.3                h3fd9d12_0    sbl
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnpp                    12.0.2.50                     0    nvidia
libntlm                   1.4               h7f98852_1002    conda-forge
libnvjitlink              12.1.105             hd3aeb46_0    conda-forge
libnvjpeg                 12.1.1.14                     0    nvidia
libpng                    1.6.39               h5eee18b_0
libpq                     12.17                hdbd6064_0
libprotobuf               3.20.3               he621ea3_0
libsodium                 1.0.18               h7b6447c_0
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               hc0a3c3a_7    conda-forge
libtasn1                  4.19.0               h5eee18b_0
libtiff                   4.5.1                h6a678d5_0
libunistring              0.9.10               h27cfd23_0
libuuid                   1.41.5               h5eee18b_0
libwebp-base              1.3.2                h5eee18b_0
libxcb                    1.15                 h7f8727e_0
libxkbcommon              1.0.1                h5eee18b_1
libxml2                   2.10.4               hfdd30dd_2
libzlib                   1.2.13               h4ab18f5_6    conda-forge
llvm-openmp               14.0.6               h9e868ea_0
lz4-c                     1.9.4                h6a678d5_1
lzo                       2.10                 h7b6447c_2
markdown                  3.4.1           py310h06a4308_0
markov_clustering         0.0.6                      py_0    bioconda
markupsafe                2.1.3           py310h5eee18b_0
matplotlib                3.8.4           py310h06a4308_0
matplotlib-base           3.8.4           py310h1128e8f_0
matplotlib-inline         0.1.6           py310h06a4308_0
mistune                   2.0.4           py310h06a4308_0
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0           py310h7f8727e_0
mkl_fft                   1.3.1           py310hd6ae3a3_0
mkl_random                1.2.2           py310h00e6091_0
mpc                       1.1.0                h10f8cd9_1
mpfr                      4.0.2                hb69a4c5_1
mpmath                    1.3.0           py310h06a4308_0
msms                      2.6.1                h9ee0642_3    bioconda
multidict                 6.0.4           py310h5eee18b_0
mysql                     5.7.20            hf484d3e_1001    conda-forge
nbclient                  0.8.0           py310h06a4308_0
nbconvert                 7.10.0          py310h06a4308_0
nbformat                  5.9.2           py310h06a4308_0
ncurses                   6.4                  h6a678d5_0
nest-asyncio              1.6.0           py310h06a4308_0
nettle                    3.7.3                hbbd107a_1
networkx                  3.2.1           py310h06a4308_0
notebook                  7.0.8           py310h06a4308_0
notebook-shim             0.2.3           py310h06a4308_0
numexpr                   2.8.4           py310h8879344_0
numpy                     1.26.4                   pypi_0    pypi
numpy-base                1.24.3          py310h8e6c178_0
oauthlib                  3.2.2           py310h06a4308_0
ocl-icd                   2.3.2                h5eee18b_1
openh264                  2.1.1                h4ff587b_0
openjpeg                  2.4.0                h3ad879b_0
openssl                   3.3.1                h4ab18f5_0    conda-forge
overrides                 7.4.0           py310h06a4308_0
packaging                 23.2            py310h06a4308_0
pandas                    2.2.2           py310hf9f9076_1    conda-forge
pandocfilters             1.5.0              pyhd3eb1b0_0
parso                     0.8.3              pyhd3eb1b0_0
pcre                      8.45                 h295c915_0
pdb2sql                   0.5.3              pyhd8ed1ab_0    conda-forge
pexpect                   4.8.0              pyhd3eb1b0_3
pillow                    10.3.0          py310h5eee18b_0
pip                       24.0            py310h06a4308_0
platformdirs              3.10.0          py310h06a4308_0
plotly                    5.19.0          py310h2f386ee_0
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
ply                       3.11            py310h06a4308_0
pooch                     1.8.2              pyhd8ed1ab_0    conda-forge
prometheus_client         0.14.1          py310h06a4308_0
prompt-toolkit            3.0.43          py310h06a4308_0
prompt_toolkit            3.0.43               hd3eb1b0_0
protobuf                  3.20.3          py310h6a678d5_0
psutil                    5.9.0           py310h5eee18b_0
ptyprocess                0.7.0              pyhd3eb1b0_2
pure_eval                 0.2.2              pyhd3eb1b0_0
py-cpuinfo                9.0.0           py310h06a4308_0
pyarrow                   16.1.0                   pypi_0    pypi
pyasn1                    0.4.8              pyhd3eb1b0_0
pyasn1-modules            0.2.8                      py_0
pycodestyle               2.12.0                   pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0
pyg                       2.5.2           py310_torch_2.1.0_cu121    pyg
pygments                  2.15.1          py310h06a4308_1
pyjwt                     2.8.0           py310h06a4308_0
pyopenssl                 23.2.0          py310h06a4308_0
pyparsing                 3.0.9           py310h06a4308_0
pyqt                      5.15.10         py310h6a678d5_0
pyqt5-sip                 12.13.0         py310h5eee18b_0
pysocks                   1.7.1           py310h06a4308_0
pytest                    8.2.2              pyhd8ed1ab_0    conda-forge
pytest-cov                5.0.0                    pypi_0    pypi
pytest-runner             6.0.1                    pypi_0    pypi
python                    3.10.14              h955ad1f_1
python-dateutil           2.9.0post0      py310h06a4308_2
python-fastjsonschema     2.16.2          py310h06a4308_0
python-json-logger        2.0.7           py310h06a4308_0
python-louvain            0.16               pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd3eb1b0_0
python_abi                3.10                    2_cp310    conda-forge
pytorch                   2.1.1           py3.10_cuda12.1_cudnn8.9.2_0    pytorch
pytorch-cluster           1.6.3           py310_torch_2.1.0_cu121    pyg
pytorch-cuda              12.1                 ha16c6d3_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytorch-scatter           2.1.2           py310_torch_2.1.0_cu121    pyg
pytorch-sparse            0.6.18          py310_torch_2.1.0_cu121    pyg
pytorch-spline-conv       1.2.2           py310_torch_2.1.0_cu121    pyg
pytz                      2024.1          py310h06a4308_0
pyyaml                    6.0.1           py310h5eee18b_0
pyzmq                     25.1.2          py310h6a678d5_0
qt-main                   5.15.2              h53bd1ea_10
re2                       2022.04.01           h295c915_0
readline                  8.2                  h5eee18b_0
referencing               0.30.2          py310h06a4308_0
requests                  2.32.2          py310h06a4308_0
requests-oauthlib         1.3.0                      py_0
retrying                  1.3.3              pyhd3eb1b0_2
rfc3339-validator         0.1.4           py310h06a4308_0
rfc3986-validator         0.1.1           py310h06a4308_0
rpds-py                   0.10.6          py310hb02cf49_0
rsa                       4.7.2              pyhd3eb1b0_1
ruff                      0.3.5           py310hab49468_0
scikit-learn              1.4.2           py310h1128e8f_1
scipy                     1.13.1          py310h93e2701_0    conda-forge
send2trash                1.8.2           py310h06a4308_0
setuptools                69.5.1          py310h06a4308_0
sip                       6.7.12          py310h6a678d5_0
six                       1.16.0             pyhd3eb1b0_1
sniffio                   1.3.0           py310h06a4308_0
soupsieve                 2.5             py310h06a4308_0
sqlite                    3.45.3               h5eee18b_0
stack_data                0.2.0              pyhd3eb1b0_0
sympy                     1.12            py310h06a4308_0
tables                    3.9.2           py310h0016290_0
tenacity                  8.2.2           py310h06a4308_0
tensorboard               2.12.1          py310h06a4308_0
tensorboard-data-server   0.7.0           py310h52d8a92_0
tensorboard-plugin-wit    1.8.1           py310h06a4308_0
terminado                 0.17.1          py310h06a4308_0
threadpoolctl             2.2.0              pyh0d69192_0
tinycss2                  1.2.1           py310h06a4308_0
tk                        8.6.14               h39e8969_0
tomli                     2.0.1           py310h06a4308_0
torchaudio                2.1.1               py310_cu121    pytorch
torchtriton               2.1.0                     py310    pytorch
torchvision               0.16.1              py310_cu121    pytorch
tornado                   6.3.3           py310h5eee18b_0
tqdm                      4.66.4          py310h2f386ee_0
traitlets                 5.7.1           py310h06a4308_0
types-python-dateutil     2.9.0.20240316           pypi_0    pypi
typing-extensions         4.11.0          py310h06a4308_0
typing_extensions         4.11.0          py310h06a4308_0
tzdata                    2024a                h04d1e81_0
unicodedata2              15.1.0          py310h5eee18b_0
uri-template              1.3.0                    pypi_0    pypi
urllib3                   2.2.1           py310h06a4308_0
wcwidth                   0.2.5              pyhd3eb1b0_0
webcolors                 24.6.0                   pypi_0    pypi
webencodings              0.5.1           py310h06a4308_1
websocket-client          1.8.0           py310h06a4308_0
werkzeug                  3.0.3           py310h06a4308_0
wheel                     0.43.0          py310h06a4308_0
xz                        5.4.6                h5eee18b_1
yaml                      0.2.5                h7b6447c_0
yarl                      1.9.3           py310h5eee18b_0
zeromq                    4.3.5                h6a678d5_0
zlib                      1.2.13               h4ab18f5_6    conda-forge
zlib-ng                   2.0.7                h5eee18b_0
zstd                      1.5.5                hc292b87_2

We have a privileged user for installations, and regular users. I noticed that writing privileges in deeprank2/ are required to at least run pytest (eg write test.pth.tar). The OS error seems independent of privileges though. I wonder if for some reason the object features_transform in features_transform_to_save = copy.deepcopy(self.features_transform) has a problem, so there is no source code to actually extract, so I am trying to figure out how to check it.

rubenalv commented 2 months ago

@gcroci2, I found the reason for the problem: I was running the code (https://github.com/DeepRank/deeprank2/blob/main/tutorials/training.ipynb) from the Python command line (not from the original Jupiter notebook). I copied it to a script and run python script.py, and the training completed as expected.

I don't know enough Python to understand why, because I could use inspect.getsource() from command line in a mock serialized/deserialized object. If you knew why and could point me to an explanation it would be great!

Perhaps it would be worth it to add a note to not run in command line in the tutorial? I think the issue is OK to be closed now.

gcroci2 commented 2 months ago

@gcroci2, I found the reason for the problem: I was running the code (https://github.com/DeepRank/deeprank2/blob/main/tutorials/training.ipynb) from the Python command line (not from the original Jupiter notebook). I copied it to a script and run python script.py, and the training completed as expected.

I don't know enough Python to understand why, because I could use inspect.getsource() from command line in a mock serialized/deserialized object. If you knew why and could point me to an explanation it would be great!

@rubenalv It sounds like the issue is related to how inspect.getsource() interacts with code executed in different environments.

If I understand correctly, the code was trowing an error about inspect.getsource() when you run the code from the Python command line. However, it worked when running the same code within a Jupyter notebook or a Python script.

Jupyter notebooks use IPython, which has mechanisms to keep track of the source code of cells. This makes inspect.getsource() work more reliably within a notebook. The notebook environment can store the cell's code in a way that inspect can access it, even if it's not coming from a traditional script file.
When you run python script.py, the script is loaded from the file system, and inspect.getsource() can locate the source code of functions and classes because it knows the exact file and location. If the function or class definition is imported from another module, inspect.getsource() can still locate it, assuming the source file is accessible.

So to conclude, my guess is that when you run the code from the Python command line you are defining an "interactive" job on the cluster without a source file, and inspect.getsource() fails because it can't locate the source code.

Perhaps it would be worth it to add a note to not run in command line in the tutorial? I think the issue is OK to be closed now.

Good suggestion, thanks :) I opened a PR (#620) for it.

rubenalv commented 2 months ago

All much appreciated, thank you for your work to make DeepRank2 accesible!

DeepRank / deeprank2

Bug: OSError: could not get source code #609