Issues with train.py script

[MKCarter]

Hi,

When running the train.py script, it looks like there is an issue with rdkit's multiprocessing of the descriptors:

python train.py --save_dir models/mike/ --compound_csv data/r_al_final_public.csv --rating_col label --val_size 0.2 --n_epochs 10 --featurizer_name morgan_count_rdkit_2d --num_workers 10
/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/utilities/seed.py:55: UserWarning: No seed found, seed set to 2600799618
  rank_zero_warn(f"No seed found, seed set to {seed}")
Global seed set to 2600799618
[2023-02-28 10:53:20,895] [INFO] [cleaners.py:111:ensure_readability_and_remove] Checking SMILES validity and removing unreadable ones...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5276/5276 [00:01<00:00, 3641.47it/s]
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: models/mike/default

  | Name | Type    | Params
---------------------------------
0 | net  | RankNet | 766 K 
---------------------------------
766 K     Trainable params
0         Non-trainable params
766 K     Total params
3.065     Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "/Users/michaelcarter/DD_tools/molskill/train.py", line 206, in <module>
    train_ranknet(
  File "/Users/michaelcarter/DD_tools/molskill/train.py", line 109, in train_ranknet
    trainer.fit(
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_train
    self._run_sanity_check()
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1411, in _run_sanity_check
    val_loop.run()
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 154, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 199, in run
    self.on_run_start(*args, **kwargs)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 87, in on_run_start
    self._data_fetcher = iter(data_fetcher)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 178, in __iter__
    self.dataloader_iter = iter(self.dataloader)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 368, in __iter__
    return self._get_iterator()
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 314, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 927, in __init__
    w.start()
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'Boost.Python.function' object

I think this might be related to this rdkit post https://github.com/rdkit/rdkit/issues/2529

Thanks

[MKCarter]

I'm also seeing this error with the score.py script:

python score.py --model_ckpt models/default/checkpoints/last.ckpt --featurizer_name rdkit_2d_norm --compound_csv test.csv --smiles_col SMILES --output_csv test_out.csv 
[2023-02-28 11:04:47,090] [INFO] [cleaners.py:111:ensure_readability_and_remove] Checking SMILES validity and removing unreadable ones...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 845.37it/s]
Traceback (most recent call last):
  File "/Users/michaelcarter/DD_tools/molskill/score.py", line 61, in <module>
    LitRankNet.load_from_checkpoint(args.model_ckpt, input_size=featurizer.dim())
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 161, in load_from_checkpoint
    model = cls._load_model_state(checkpoint, strict=strict, **kwargs)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/pytorch_lightning/core/saving.py", line 209, in _load_model_state
    keys = model.load_state_dict(checkpoint["state_dict"], strict=strict)
  File "/Users/michaelcarter/opt/miniconda3/envs/molskill/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1497, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LitRankNet:
    size mismatch for net.encoder.0.weight: copying a param with shape torch.Size([256, 2220]) from checkpoint, the shape in current model is torch.Size([256, 172]).

I thought it may have been the --featurizer_name but it throws the same error for all featurizers.

[MKCarter]

The output from pytest also confirms my comment above:

FAILED tests/test_scorer.py::test_scorer[1] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_scorer.py::test_scorer[25] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[rdkit_2d] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[rdkit_2d_norm] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan_count] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[avalon] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[avalon_count] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d] - TypeError: cannot pickle 'Boost.Python.function' object
FAILED tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d_norm] - TypeError: cannot pickle 'Boost.Python.function' object

[josejimenezluna]

Hi @MKCarter

Many thanks for using our software :)

Could you provide a few more details on where you're running our code? Does this happen with the conda package or with the environment file? If so, are you using a fresh environment?

[MKCarter]

Hi @josejimenezluna

I am not using the conda package as I am running on an M1 Mac, I made the env using:

conda create -n molskills python=3.9

I then pip installed all the requirements from the environment.yml

I think this error is caused by parallelisation. From what I have been told C++ boost functions surfaced as Python import can't be pickled by pickle , which is used to pass them around the processes. If the code is using multiprocessing you can try swapping that with multiprocess, which uses dill instead of pickle and can handle Boost functions. The modules are exactly the same, so it's just a matter of swapping the module you are importing.

Cheers

[MKCarter]

If I do try to conda install this is the error I see BTW:

conda install molskill -c msr-ai4science -c conda-forge

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - molskill

Current channels:

  - https://conda.anaconda.org/msr-ai4science/osx-64
  - https://conda.anaconda.org/msr-ai4science/noarch
  - https://conda.anaconda.org/conda-forge/osx-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://conda.anaconda.org/omnia/osx-64
  - https://conda.anaconda.org/omnia/noarch
  - https://repo.anaconda.com/pkgs/main/osx-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-64
  - https://repo.anaconda.com/pkgs/r/noarch

[josejimenezluna]

Hi @MKCarter. We currently only support Linux 3.9 builds (we state this in the README.md file). I cannot reproduce your issue on a clean env on a Linux machine:

(base) REDMOND.jjimenezluna@XXX:~/molskill$ mamba env create -n molskill-test -f environment.yml python=3.9
pytorch/linux-64         [====================] (00m:00s) No change
pytorch/noarch           [====================] (00m:00s) Done
pkgs/r/noarch            [====================] (00m:00s) No change
pkgs/r/linux-64          [====================] (00m:00s) No change
pkgs/main/noarch         [====================] (00m:00s) Done
pkgs/main/linux-64       [====================] (00m:00s) Done
conda-forge/noarch       [====================] (00m:02s) Done
conda-forge/linux-64     [====================] (00m:05s) Done

Looking for: ['python=3.9', 'pip=22.1.2', 'numpy=1.23.1', 'pandas=1.4.2', 'scipy=1.8.1', 'rdkit=2022.03.3', 'tqdm=4.64.0', 'scikit-learn=1.0.1', 'pytorch=1.11', 'cudatoolkit=11.3', 'black=22.6.0', 'pre-commit=2.20.0', 'pyright=1.1.269']

Transaction

  Prefix: /home/jjimenezluna/miniconda3/envs/molskill-test

  Updating specs:

   - python=3.9
   - pip=22.1.2
   - numpy=1.23.1
   - pandas=1.4.2
   - scipy=1.8.1
   - rdkit=2022.03.3
   - tqdm=4.64.0
   - scikit-learn=1.0.1
   - pytorch=1.11
   - cudatoolkit=11.3
   - black=22.6.0
   - pre-commit=2.20.0
   - pyright=1.1.269

  Package                        Version  Build                    Channel                    Size
────────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex                    0.1  conda_forge              conda-forge/linux-64     Cached
  + _openmp_mutex                    4.5  2_gnu                    conda-forge/linux-64     Cached
  + black                         22.6.0  py39hf3d152e_2           conda-forge/linux-64     Cached
  + boost                         1.74.0  py39h5472131_5           conda-forge/linux-64     Cached
  + boost-cpp                     1.74.0  h75c5d50_8               conda-forge/linux-64     Cached
  + brotli                         1.0.9  h166bdaf_8               conda-forge/linux-64     Cached
  + brotli-bin                     1.0.9  h166bdaf_8               conda-forge/linux-64     Cached
  + bzip2                          1.0.8  h7f98852_4               conda-forge/linux-64     Cached
  + ca-certificates            2022.12.7  ha878542_0               conda-forge/linux-64     Cached
  + cairo                         1.16.0  ha61ee94_1014            conda-forge/linux-64     Cached
  + certifi                    2022.12.7  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + cffi                          1.15.1  py39he91dace_3           conda-forge/linux-64     Cached
  + cfgv                           3.3.1  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + click                          8.1.3  py39hf3d152e_1           conda-forge/linux-64     Cached
  + colorama                       0.4.6  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + contourpy                      1.0.7  py39h4b4f3f3_0           conda-forge/linux-64     210 KB
  + cudatoolkit                   11.3.1  h9edb442_11              conda-forge/linux-64     Cached
  + cudnn                       8.4.1.50  hed8a83a_0               conda-forge/linux-64     Cached
  + cycler                        0.11.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + distlib                        0.3.6  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + expat                          2.5.0  h27087fc_0               conda-forge/linux-64     Cached
  + filelock                       3.9.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + font-ttf-dejavu-sans-mono       2.37  hab24e00_0               conda-forge/noarch       Cached
  + font-ttf-inconsolata           3.000  h77eed37_0               conda-forge/noarch       Cached
  + font-ttf-source-code-pro       2.038  h77eed37_0               conda-forge/noarch       Cached
  + font-ttf-ubuntu                 0.83  hab24e00_0               conda-forge/noarch       Cached
  + fontconfig                    2.14.2  h14ed4e7_0               conda-forge/linux-64     Cached
  + fonts-conda-ecosystem              1  0                        conda-forge/noarch       Cached
  + fonts-conda-forge                  1  0                        conda-forge/noarch       Cached
  + fonttools                     4.38.0  py39hb9d737c_1           conda-forge/linux-64     Cached
  + freetype                      2.12.1  hca18f0e_1               conda-forge/linux-64     Cached
  + gettext                       0.21.1  h27087fc_0               conda-forge/linux-64     Cached
  + greenlet                       2.0.2  py39h227be39_0           conda-forge/linux-64     186 KB
  + icu                             70.1  h27087fc_0               conda-forge/linux-64     Cached
  + identify                      2.5.18  pyhd8ed1ab_0             conda-forge/noarch        76 KB
  + importlib-metadata             6.0.0  pyha770c72_0             conda-forge/noarch       Cached
  + importlib-resources           5.12.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + importlib_resources           5.12.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + intel-openmp                2022.1.0  h9e868ea_3769            pkgs/main/linux-64       Cached
  + joblib                         1.2.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + jpeg                              9e  h0b41bf4_3               conda-forge/linux-64     Cached
  + kiwisolver                     1.4.4  py39hf939315_1           conda-forge/linux-64     Cached
  + lcms2                           2.14  hfd0df8a_1               conda-forge/linux-64     Cached
  + ld_impl_linux-64                2.40  h41732ed_0               conda-forge/linux-64     Cached
  + lerc                           4.0.0  h27087fc_0               conda-forge/linux-64     Cached
  + libblas                        3.9.0  16_linux64_mkl           conda-forge/linux-64     Cached
  + libbrotlicommon                1.0.9  h166bdaf_8               conda-forge/linux-64     Cached
  + libbrotlidec                   1.0.9  h166bdaf_8               conda-forge/linux-64     Cached
  + libbrotlienc                   1.0.9  h166bdaf_8               conda-forge/linux-64     Cached
  + libcblas                       3.9.0  16_linux64_mkl           conda-forge/linux-64     Cached
  + libdeflate                      1.17  h0b41bf4_0               conda-forge/linux-64     Cached
  + libffi                         3.4.2  h7f98852_5               conda-forge/linux-64     Cached
  + libgcc-ng                     12.2.0  h65d4601_19              conda-forge/linux-64     Cached
  + libgfortran-ng                12.2.0  h69a702a_19              conda-forge/linux-64     Cached
  + libgfortran5                  12.2.0  h337968e_19              conda-forge/linux-64     Cached
  + libglib                       2.74.1  h606061b_1               conda-forge/linux-64     Cached
  + libgomp                       12.2.0  h65d4601_19              conda-forge/linux-64     Cached
  + libiconv                        1.17  h166bdaf_0               conda-forge/linux-64     Cached
  + liblapack                      3.9.0  16_linux64_mkl           conda-forge/linux-64     Cached
  + libnsl                         2.0.0  h7f98852_0               conda-forge/linux-64     Cached
  + libpng                        1.6.39  h753d276_0               conda-forge/linux-64     Cached
  + libprotobuf                   3.20.2  h6239696_0               conda-forge/linux-64     Cached
  + libsqlite                     3.40.0  h753d276_0               conda-forge/linux-64     Cached
  + libstdcxx-ng                  12.2.0  h46fd767_19              conda-forge/linux-64     Cached
  + libtiff                        4.5.0  h6adf6a1_2               conda-forge/linux-64     Cached
  + libuuid                       2.32.1  h7f98852_1000            conda-forge/linux-64     Cached
  + libuv                         1.44.2  h166bdaf_0               conda-forge/linux-64     Cached
  + libwebp-base                   1.2.4  h166bdaf_0               conda-forge/linux-64     Cached
  + libxcb                          1.13  h7f98852_1004            conda-forge/linux-64     Cached
  + libzlib                       1.2.13  h166bdaf_4               conda-forge/linux-64     Cached
  + magma                          2.5.4  hc72dce7_4               conda-forge/linux-64     Cached
  + matplotlib-base                3.7.0  py39he190548_0           conda-forge/linux-64       6 MB
  + mkl                         2022.1.0  hc2b9512_224             pkgs/main/linux-64       Cached
  + munkres                        1.1.4  pyh9f0ad1d_0             conda-forge/noarch       Cached
  + mypy_extensions                1.0.0  pyha770c72_0             conda-forge/noarch       Cached
  + nccl                        2.14.3.1  h0800d71_0               conda-forge/linux-64     Cached
  + ncurses                          6.3  h27087fc_1               conda-forge/linux-64     Cached
  + ninja                         1.11.1  h924138e_0               conda-forge/linux-64       2 MB
  + nodeenv                        1.7.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + nodejs                       18.13.0  h8d033a5_0               conda-forge/linux-64      14 MB
  + numpy                         1.23.1  py39hba7629e_0           conda-forge/linux-64     Cached
  + openjpeg                       2.5.0  hfec8fc6_2               conda-forge/linux-64     Cached
  + openssl                        3.0.8  h0b41bf4_0               conda-forge/linux-64     Cached
  + packaging                       23.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + pandas                         1.4.2  py39h1832856_2           conda-forge/linux-64     Cached
  + pathspec                      0.11.0  pyhd8ed1ab_0             conda-forge/noarch        36 KB
  + pcre2                          10.40  hc3806b6_0               conda-forge/linux-64     Cached
  + pillow                         9.4.0  py39h2320bf1_1           conda-forge/linux-64      44 MB
  + pip                           22.1.2  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + pixman                        0.40.0  h36c2ea0_0               conda-forge/linux-64     Cached
  + platformdirs                   3.0.0  pyhd8ed1ab_0             conda-forge/noarch        17 KB
  + pre-commit                    2.20.0  py39hf3d152e_1           conda-forge/linux-64     Cached
  + pthread-stubs                    0.4  h36c2ea0_1001            conda-forge/linux-64     Cached
  + pycairo                       1.23.0  py39h23c5bb2_0           conda-forge/linux-64     Cached
  + pycparser                       2.21  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + pyparsing                      3.0.9  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + pyright                      1.1.269  py39hb9d737c_0           conda-forge/linux-64     Cached
  + python                        3.9.16  h2782a2a_0_cpython       conda-forge/linux-64      23 MB
  + python-dateutil                2.8.2  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + python_abi                       3.9  3_cp39                   conda-forge/linux-64     Cached
  + pytorch                       1.11.0  cuda112py39ha0cca9b_202  conda-forge/linux-64     Cached
  + pytz                        2022.7.1  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + pyyaml                           6.0  py39hb9d737c_5           conda-forge/linux-64     Cached
  + rdkit                      2022.03.3  py39h89e00b9_0           conda-forge/linux-64     Cached
  + readline                       8.1.2  h0f457ee_0               conda-forge/linux-64     Cached
  + reportlab                     3.6.12  py39h8bbe095_2           conda-forge/linux-64     Cached
  + scikit-learn                   1.0.1  py39h4dfa638_3           conda-forge/linux-64     Cached
  + scipy                          1.8.1  py39hddc5342_3           conda-forge/linux-64     Cached
  + setuptools                    67.4.0  pyhd8ed1ab_0             conda-forge/noarch       567 KB
  + six                           1.16.0  pyh6c4a22f_0             conda-forge/noarch       Cached
  + sleef                          3.5.1  h9b69904_2               conda-forge/linux-64     Cached
  + sqlalchemy                     2.0.4  py39h72bdee0_0           conda-forge/linux-64       2 MB
  + threadpoolctl                  3.1.0  pyh8a188c0_0             conda-forge/noarch       Cached
  + tk                            8.6.12  h27826a3_0               conda-forge/linux-64     Cached
  + toml                          0.10.2  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + tomli                          2.0.1  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + tqdm                          4.64.0  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + typing-extensions              4.4.0  hd8ed1ab_0               conda-forge/noarch       Cached
  + typing_extensions              4.4.0  pyha770c72_0             conda-forge/noarch       Cached
  + tzdata                         2022g  h191b570_0               conda-forge/noarch       Cached
  + ukkonen                        1.0.1  py39hf939315_3           conda-forge/linux-64     Cached
  + unicodedata2                  15.0.0  py39hb9d737c_0           conda-forge/linux-64     Cached
  + virtualenv                   20.19.0  pyhd8ed1ab_0             conda-forge/noarch         6 MB
  + wheel                         0.38.4  pyhd8ed1ab_0             conda-forge/noarch       Cached
  + xorg-kbproto                   1.0.7  h7f98852_1002            conda-forge/linux-64     Cached
  + xorg-libice                   1.0.10  h7f98852_0               conda-forge/linux-64     Cached
  + xorg-libsm                     1.2.3  hd9c2040_1000            conda-forge/linux-64     Cached
  + xorg-libx11                    1.7.2  h7f98852_0               conda-forge/linux-64     Cached
  + xorg-libxau                    1.0.9  h7f98852_0               conda-forge/linux-64     Cached
  + xorg-libxdmcp                  1.1.3  h7f98852_0               conda-forge/linux-64     Cached
  + xorg-libxext                   1.3.4  h0b41bf4_2               conda-forge/linux-64     Cached
  + xorg-libxrender               0.9.10  h7f98852_1003            conda-forge/linux-64     Cached
  + xorg-renderproto              0.11.1  h7f98852_1002            conda-forge/linux-64     Cached
  + xorg-xextproto                 7.3.0  h0b41bf4_1003            conda-forge/linux-64     Cached
  + xorg-xproto                   7.0.31  h7f98852_1007            conda-forge/linux-64     Cached
  + xz                             5.2.6  h166bdaf_0               conda-forge/linux-64     Cached
  + yaml                           0.2.5  h7f98852_2               conda-forge/linux-64     Cached
  + zipp                          3.15.0  pyhd8ed1ab_0             conda-forge/noarch        17 KB
  + zlib                          1.2.13  h166bdaf_4               conda-forge/linux-64     Cached
  + zstd                           1.5.2  h3eb15da_6               conda-forge/linux-64     Cached

  Summary:

  Install: 140 packages

  Total download: 100 MB

────────────────────────────────────────────────────────────────────────────────────────────────────

Downloading and Extracting Packages
setuptools-67.4.0    | 567 KB    | ####################################################### | 100%
ninja-1.11.1         | 2.1 MB    | ####################################################### | 100%
python-3.9.16        | 23.0 MB   | ####################################################### | 100%
pathspec-0.11.0      | 36 KB     | ####################################################### | 100%
platformdirs-3.0.0   | 17 KB     | ####################################################### | 100%
contourpy-1.0.7      | 210 KB    | ####################################################### | 100%
nodejs-18.13.0       | 14.4 MB   | ####################################################### | 100%
pillow-9.4.0         | 44.1 MB   | ####################################################### | 100%
identify-2.5.18      | 76 KB     | ####################################################### | 100%
virtualenv-20.19.0   | 6.0 MB    | ####################################################### | 100%
greenlet-2.0.2       | 186 KB    | ####################################################### | 100%
matplotlib-base-3.7. | 6.4 MB    | ####################################################### | 100%
zipp-3.15.0          | 17 KB     | ####################################################### | 100%
sqlalchemy-2.0.4     | 2.4 MB    | ####################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: / By downloading and using the CUDA Toolkit conda packages, you accept the terms and conditions of the CUDA End User License Agreement (EULA): https://docs.nvidia.com/cuda/eula/index.html

\ By downloading and using the cuDNN conda packages, you accept the terms and conditions of the NVIDIA cuDNN EULA -
  https://docs.nvidia.com/deeplearning/cudnn/sla/index.html

done
Installing pip dependencies: | Ran pip subprocess with arguments:
['/home/jjimenezluna/miniconda3/envs/molskill-test/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt']
Pip subprocess output:
Collecting pytorch-lightning==1.6.3
  Using cached pytorch_lightning-1.6.3-py3-none-any.whl (584 kB)
Collecting torchmetrics>=0.4.1
  Downloading torchmetrics-0.11.1-py3-none-any.whl (517 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 517.2/517.2 kB 2.2 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.57.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (4.64.0)
Collecting pyDeprecate<0.4.0,>=0.3.1
  Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
Requirement already satisfied: packaging>=17.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (23.0)
Requirement already satisfied: torch>=1.8.* in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (1.11.0.post202)
Requirement already satisfied: PyYAML>=5.4 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (6.0)
Requirement already satisfied: typing-extensions>=4.0.0 in /home/jjimenezluna/.local/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (4.3.0)
Collecting fsspec[http]!=2021.06.0,>=2021.05.0
  Downloading fsspec-2023.1.0-py3-none-any.whl (143 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.0/143.0 kB 18.6 MB/s eta 0:00:00
Collecting tensorboard>=2.2.0
  Downloading tensorboard-2.12.0-py3-none-any.whl (5.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 25.9 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.17.2 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (1.23.1)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1
  Downloading aiohttp-3.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 59.9 MB/s eta 0:00:00
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting protobuf>=3.19.6
  Downloading protobuf-4.22.0-cp37-abi3-manylinux2014_x86_64.whl (302 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.4/302.4 kB 34.5 MB/s eta 0:00:00
Collecting markdown>=2.6.8
  Using cached Markdown-3.4.1-py3-none-any.whl (93 kB)
Collecting google-auth<3,>=1.6.3
  Downloading google_auth-2.16.1-py2.py3-none-any.whl (177 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.2/177.2 kB 26.8 MB/s eta 0:00:00
Collecting grpcio>=1.48.2
  Downloading grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 43.7 MB/s eta 0:00:00
Collecting tensorboard-data-server<0.8.0,>=0.7.0
  Downloading tensorboard_data_server-0.7.0-py3-none-manylinux2014_x86_64.whl (6.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 43.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=41.0.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (67.4.0)
Collecting werkzeug>=1.0.1
  Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 32.4 MB/s eta 0:00:00
Requirement already satisfied: wheel>=0.26 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (0.38.4)
Collecting absl-py>=0.4
  Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
Collecting tensorboard-plugin-wit>=1.6.0
  Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
Collecting attrs>=17.3.0
  Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting async-timeout<5.0,>=4.0.0a3
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting charset-normalizer<4.0,>=2.0
  Using cached charset_normalizer-3.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.9-py3-none-any.whl (34 kB)
Requirement already satisfied: six>=1.9.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (1.16.0)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<6.0,>=2.0.0
  Downloading cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting requests-oauthlib>=0.7.0
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: importlib-metadata>=4.4 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (6.0.0)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Requirement already satisfied: certifi>=2017.4.17 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (2022.12.7)
Collecting MarkupSafe>=2.1.1
  Downloading MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Requirement already satisfied: zipp>=0.5 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning==1.6.3->-r /home/jjimenezluna/molskill/condaenv.3vept7ab.requirements.txt (line 1)) (3.15.0)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
Installing collected packages: tensorboard-plugin-wit, pyasn1, charset-normalizer, urllib3, tensorboard-data-server, rsa, pyDeprecate, pyasn1-modules, protobuf, oauthlib, multidict, MarkupSafe, idna, grpcio, fsspec, frozenlist, cachetools, attrs, async-timeout, absl-py, yarl, werkzeug, torchmetrics, requests, markdown, google-auth, aiosignal, requests-oauthlib, aiohttp, google-auth-oauthlib, tensorboard, pytorch-lightning
Successfully installed MarkupSafe-2.1.2 absl-py-1.4.0 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 attrs-22.2.0 cachetools-5.3.0 charset-normalizer-3.0.1 frozenlist-1.3.3 fsspec-2023.1.0 google-auth-2.16.1 google-auth-oauthlib-0.4.6 grpcio-1.51.3 idna-3.4 markdown-3.4.1 multidict-6.0.4 oauthlib-3.2.2 protobuf-4.22.0 pyDeprecate-0.3.2 pyasn1-0.4.8 pyasn1-modules-0.2.8 pytorch-lightning-1.6.3 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 tensorboard-2.12.0 tensorboard-data-server-0.7.0 tensorboard-plugin-wit-1.8.1 torchmetrics-0.11.1 urllib3-1.26.14 werkzeug-2.2.3 yarl-1.8.2

done
#
# To activate this environment, use
#
#     $ conda activate molskill-test
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) REDMOND.jjimenezluna@XXX:~/molskill$ conda activate molskill-test
(molskill-test) REDMOND.jjimenezluna@XXX:~/molskill$ pip install pytest
Collecting pytest
  Downloading pytest-7.2.1-py3-none-any.whl (317 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 kB 1.6 MB/s eta 0:00:00
Collecting iniconfig
  Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Requirement already satisfied: attrs>=19.2.0 in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytest) (22.2.0)
Collecting exceptiongroup>=1.0.0rc8
  Using cached exceptiongroup-1.1.0-py3-none-any.whl (14 kB)
Requirement already satisfied: tomli>=1.0.0 in /home/jjimenezluna/.local/lib/python3.9/site-packages (from pytest) (2.0.1)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: packaging in /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages (from pytest) (23.0)
Installing collected packages: pluggy, iniconfig, exceptiongroup, pytest
Successfully installed exceptiongroup-1.1.0 iniconfig-2.0.0 pluggy-1.0.0 pytest-7.2.1
(molskill-test) REDMOND.jjimenezluna@XXX:~/molskill$ export PYTHONPATH=.
(molskill-test) REDMOND.jjimenezluna@XXX:~/molskill$ pytest
====================================== test session starts =======================================
platform linux -- Python 3.9.16, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/jjimenezluna/molskill
collected 21 items

tests/test_featurizers.py ..........                                                       [ 47%]
tests/test_links.py .                                                                      [ 52%]
tests/test_scorer.py ..                                                                    [ 61%]
tests/test_train.py ........                                                               [100%]

======================================== warnings summary ========================================
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pkg_resources/__init__.py:2804: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4
../miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/utils/tensorboard/__init__.py:4: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):

../miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/__init__.py:38
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/__init__.py:38: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pytorch_lightning')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    __import__("pkg_resources").declare_namespace(__name__)

../miniconda3/envs/molskill-test/lib/python3.9/site-packages/sklearn/utils/multiclass.py:14
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/sklearn/utils/multiclass.py:14: DeprecationWarning: Please use `spmatrix` from the `scipy.sparse` namespace, the `scipy.sparse.base` namespace is deprecated.
    from scipy.sparse.base import spmatrix

tests/test_scorer.py::test_scorer[1]
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/torch/distributed/_sharded_tensor/__init__.py:8: DeprecationWarning: torch.distributed._sharded_tensor will be deprecated, use torch.distributed._shard.sharded_tensor instead
    warnings.warn(

tests/test_train.py::test_loss_goes_down[rdkit_2d]
tests/test_train.py::test_loss_goes_down[rdkit_2d_norm]
tests/test_train.py::test_loss_goes_down[morgan]
tests/test_train.py::test_loss_goes_down[morgan_count]
tests/test_train.py::test_loss_goes_down[avalon]
tests/test_train.py::test_loss_goes_down[avalon_count]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d_norm]
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:133: UserWarning: You defined a `validation_step` but have no `val_dataloader`. Skipping val loop.
    rank_zero_warn("You defined a `validation_step` but have no `val_dataloader`. Skipping val loop.")

tests/test_train.py::test_loss_goes_down[rdkit_2d]
tests/test_train.py::test_loss_goes_down[rdkit_2d_norm]
tests/test_train.py::test_loss_goes_down[morgan]
tests/test_train.py::test_loss_goes_down[morgan_count]
tests/test_train.py::test_loss_goes_down[avalon]
tests/test_train.py::test_loss_goes_down[avalon_count]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d]
tests/test_train.py::test_loss_goes_down[morgan_count_rdkit_2d_norm]
  /home/jjimenezluna/miniconda3/envs/molskill-test/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1927: PossibleUserWarning: The number of training batches (2) is smaller than the logging interval Trainer(log_every_n_steps=20). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
    rank_zero_warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ 21 passed, 23 warnings in 40.79s ================================

I believe your issue lies on how the multiprocessing is handled by the torch DataLoader across different platforms.

Can you manually change the num_workers=1 arg in the MolSkillScorer instance and see whether it works?

[MKCarter]

@josejimenezluna I tried, but no luck! Thanks for the suggestion and the help.

I will take a look and see if there is anyway to make this Mac compatible.

If I find a solution, I will update you.

Cheers

[MKCarter]

Thinking about it, the easiest solution is to use docker...

[josejimenezluna]

Hi @MKCarter. I am closing this since we do not currently plan on providing OS X support.

[kjelljorner]

An easy (inefficient) fix for this is to set scorer = MolSkillScorer(num_workers=0), which disables multiprocessing.