Closed marcosfelt closed 2 years ago
Hi,
Could you provide more information about the Python environment you use? In addition, what environment do you use to run the tutorial, i.e. notebook, ipython/python terminal?
I used pip install .
inside a python virtual environment ( created with python3 -m venv .venv
). I had to install rdkit manually using pip install rdkit_pypi
. I was running from the python repl to just to test the tutorial.
I've created a similar environment on my Linux workstation.
python -m venv test_venv
python -m pip install --upgrade pip
rdkit
, I tried both python -m pip install rdkit
and python -m pip install rdkit-pypi
(I didn't notice any difference)rxnutils
with python -m pip install .
ipython
terminal and run the same tutorial code with no issuespoetry
with python -m pip install poetry
poetry install
(this downgraded a lot of packages and installed a few)ipython
terminal and run the same tutorial code with no issuesLooking at the error message again, after calling timeout_decorator
it jumps out of your virtualenv and calls multiprocessing
from homebrew
, not sure if this is intended or not.
I would suggest to recreate your virtualenv as follows and test again.
python -m venv --clear <venv_name> # Python >= 3.8 works
. <venv_name>/bin/activate
python -m pip install --upgrade pip
python -m pip install rdkit or python -m pip install rdkit-pypi
python -m pip install poetry
# Navigate to the root of the repository and run
poetry install
Or you can just install rxnutils
directly from GitHub as follows:
python -m venv --clear <venv_name> # Python >= 3.8 works
. <venv_name>/bin/activate
python -m pip install --upgrade pip
python -m pip install rdkit or python -m pip install rdkit-pypi
python -m pip install git+https://github.com/MolecularAI/reaction_utils
You should then have rxnutils
installed in the virtualenv named <venv_name>
.
You can use python
, ipython
and jupyter notebook
from the virtualenv.
Thanks for checking this in detail! I just did some digging and it looks like this is a problem with the timeout_decorator
library. You can see the open issue here; I'm guessing it fails on some platforms and works fine on others. I'd recommend maybe getting rid of the dependency on timeout-decorator as it seems it's not updated regularly; there's also this fork that seems more up to date: https://github.com/bitranox/wrapt_timeout_decorator
Also it's not best practice, to require people to use poetry to install the package (and trust me I love poetry!). Would it be possible to upload to pypi? Poetry has a nice command for doing this: https://python-poetry.org/docs/libraries/#publishing-to-pypi
Thanks for checking this in detail! I just did some digging and it looks like this is a problem with the
timeout_decorator
library. You can see the open issue here; I'm guessing it fails on some platforms and works fine on others. I'd recommend maybe getting rid of the dependency on timeout-decorator as it seems it's not updated regularly; there's also this fork that seems more up to date: https://github.com/bitranox/wrapt_timeout_decorator
Thanks, we'll have to look into it further. It seems wrapt_timeout_decorator is a better alternative.
Also it's not best practice, to require people to use poetry to install the package (and trust me I love poetry!). Would it be possible to upload to pypi? Poetry has a nice command for doing this: https://python-poetry.org/docs/libraries/#publishing-to-pypi
Can you open another issue for this?
I looked into how timeout_decorator and wrapt_timeout_decorator work.
timeout_decorator, relies only on build-in libraries, multiprocessing
and signal
for the two different ways of managing a wrapped function (using functools.wraps
).
The approach using signal
seems to only work on Linux like systems.
The approach using multiprocessing
is a more general one, which relies on the build-in multiprocessing
and pickle
libraries.
~On the other hand wrapt_timeout_decorator uses 3rd party library wrapt for wrapping functions, signal
and multiprocessing
for managing wrapped functions and dill to track issues with unpickable objects. They don't replace pickle
with dill
, only pathos
is doing that.~
Update: I was wrong. wrapt_timeout_decorator uses:
signal
,We use timeout_decorator with multiprocessing
, which on Linux uses fork to run the wrapped function while for MacOS/Windows uses spawn by default.
With forking the child process inherits state & variables from parent while with spawning the child process starts in a new Python interpreter and module is reimported and variables are recreated.
Which were the issue with pickling lies, as when spawning the import path is different than the path the function has before spawning. (https://peaku.co/questions/3030-bifurcacion-de-multiprocesamiento-()-vs-spawn-() & https://britishgeologicalsurvey.github.io/science/python-forking-vs-spawn/#:~:text=Forking%20and%20spawning%20are%20two,they%20were%20in%20the%20parent.)
We'll look for a solution.
We will switch to the wrapt_timeout_decorator package. An update will come soon.
Fixed in merged PR. Will be included in a feature release
I tried the following from the templates tutorial using python 3.8 (Mac OS Monterrey) and got a pickle error. Any ideas why?