Setting up pycalibration with pyjulia

Jordy-VL commented 3 years ago

Hi David,

Thanks for the great work! I'm finally getting around to testing your implementations, but I experience some issues installing pycalibration. I hope documenting my process of getting pycalibration installed helps future users attempt it as well :+1:

Action:

import pycalibration

Bug:


Julia executable:
    julia
Python interpreter and libpython used by PyCall.jl:
    /home/jordy/.virtualenvs/SOTA/bin/python3
    /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
Python interpreter used to import PyJulia and its libpython.
    /usr/bin/python3
    /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0

Your Python interpreter "/usr/bin/python3"
is statically linked to libpython.  Currently, PyJulia does not fully
support such Python interpreter.

The easiest workaround is to pass `compiled_modules=False` to `Julia`
constructor.  To do so, first *reboot* your Python REPL (if this happened
inside an interactive session) and then evaluate:

    >>> from julia.api import Julia
    >>> jl = Julia(compiled_modules=False)

Another workaround is to run your Python script with `python-jl`
command bundled in PyJulia.  You can simply do:

    $ python-jl PATH/TO/YOUR/SCRIPT.py

See `python-jl --help` for more information.

For more information, see:

    https://pyjulia.readthedocs.io/en/latest/troubleshooting.html

I know this is a very noob question (first-time Julia user here :p), but what is my best course of action?

I correctly installed julia and pyjulia, and if I use the suggested "compile_modules=False" workaround I can import pycalibration.

However, then it complains in the installation of pycalibration:

>>> jl = Julia(compiled_modules=False)
>>> import pycalibration
>>> pycalibration.install()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jordy/.local/lib/python3.8/site-packages/pycalibration/__init__.py", line 10, in install
    Main.include(os.path.join(script_dir, "install.jl"))
SystemError: <PyCall.jlwrap (in a Julia function called from Python)
JULIA: SystemError: opening file "/home/jordy/.local/lib/python3.8/site-packages/pycalibration/install.jl": No such file or directory

I can import specific functions, yet then there are missing dependencies, probably requiring the install():

ImportError: CalibrationErrorsDistributions not found

It seems in the python installation, the ".jl" files do not get copied to the right path:

$ ls /home/jordy/.local/lib/python3.8/site-packages/pycalibration/
calerrors.py  caltests.py  __init__.py  __pycache__  tests

As an easy fix I just grab'em and put them in the expected location:

get https://raw.githubusercontent.com/devmotion/pycalibration/main/pycalibration/install.jl ; mv install.jl /home/jordy/.local/lib/python3.8/site-packages/pycalibration/
get https://raw.githubusercontent.com/devmotion/pycalibration/main/pycalibration/setup.jl ; mv setup.jl /home/jordy/.local/lib/python3.8/site-packages/pycalibration/

Then pycalibration.install() works as a charm, as confirmed by running tests.

Final note: importing from pycalibration import calerrors as ce takes around 20 seconds, is this normal, probably due to the compiled_modules=False?

devmotion commented 3 years ago

Hi Jordy,

Thanks for opening the issue! It seems there are two different unrelated problems here:

The error when you ran import pycalibration was caused by a statically linked Python interpreter. Unfortunately, this is a fundamental limitation of PyJulia and not specific to pycalibration: https://pyjulia.readthedocs.io/en/latest/troubleshooting.html#your-python-interpreter-is-statically-linked-to-libpython I haven't experienced this issue myself since I use a different Python setup (and mostly use Julia instead :stuck_out_tongue:), so I can't provide any advice based on personal experience. It seems the easiest fix is to turn off the precompilation cache, exactly as you did. As you noticed when you ran from pycalibration import calerrors as ce, however, this will slow down loading and using Julia packages. Usually, Julia already precompiles packages during installation which leads to faster loading times. If you want to avoid this, you could build your own Python that is not statically linked: https://pyjulia.readthedocs.io/en/latest/troubleshooting.html#ultimate-fix-build-your-own-python

The error when you ran pycalibration.install() is a bug. It seems I forgot to include the Julia files in https://github.com/devmotion/pycalibration/blob/main/MANIFEST.in. I'll add them and then hopefully it works correctly. I am a bit surprised though that the tests passed and therefore it went unnoticed.

devmotion commented 3 years ago

The bug should be fixed in the master branch, Julia files should be installed correctly now.

devmotion / pycalibration

Setting up pycalibration with pyjulia #4