search

observingClouds / xbitinfo

Python wrapper of BitInformation.jl to easily compress xarray datasets based on their information content
https://xbitinfo.readthedocs.io
MIT License
53 stars 22 forks source link

Fail to install properly from PyPi #118

Closed observingClouds closed 2 years ago

observingClouds commented 2 years ago

Description

I was trying to install xbitinfo via pip but got the following error message when importing it via import xbitinfo:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./mambaforge/envs/114/lib/python3.9/site-packages/xbitinfo/__init__.py", line 4, in <module>
    from .bitround import jl_bitround, xr_bitround
  File "./mambaforge/envs/114/lib/python3.9/site-packages/xbitinfo/bitround.py", line 4, in <module>
    from .xbitinfo import _jl_bitround, get_keepbits
  File "./mambaforge/envs/114/lib/python3.9/site-packages/xbitinfo/xbitinfo.py", line 19, in <module>
    jl.using("BitInformation")
  File "./mambaforge/envs/114/lib/python3.9/site-packages/julia/core.py", line 644, in using
    self.eval("using %s" % module)
  File "./mambaforge/envs/114/lib/python3.9/site-packages/julia/core.py", line 621, in eval
    ans = self._call(src)
  File "./mambaforge/envs/114/lib/python3.9/site-packages/julia/core.py", line 549, in _call
    self.check_exception(src)
  File "./mambaforge/envs/114/lib/python3.9/site-packages/julia/core.py", line 603, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'ArgumentError: Package BitInformation not found in current path:
- Run `import Pkg; Pkg.add("BitInformation")` to install the BitInformation package.
' occurred while calling julia code:
using BitInformation

Might be related to https://stackoverflow.com/questions/15853058/run-custom-task-when-call-pip-install

What I did

pip install xbitinfo
python
> import xbitinfo

returns the error above.

pip install git+https://github.com/observingClouds/xbitinfo.git
python
> import xbitinfo

works.

aaronspring commented 2 years ago

Could you add the return from both pip install commands?

observingClouds commented 2 years ago

Here you go:

% pip install git+https://github.com/observingClouds/xbitinfo.git
Collecting git+https://github.com/observingClouds/xbitinfo.git
  Cloning https://github.com/observingClouds/xbitinfo.git to /private/var/folders/fj/g0x4n_f15tb6zfwjhzc8gvzr0000gn/T/pip-req-build-tbn7nkvw
  Running command git clone --filter=blob:none --quiet https://github.com/observingClouds/xbitinfo.git /private/var/folders/fj/g0x4n_f15tb6zfwjhzc8gvzr0000gn/T/pip-req-build-tbn7nkvw
  Resolved https://github.com/observingClouds/xbitinfo.git to commit 4d5563e3416d61b26589abcdd32a08dd72fd361b
  Preparing metadata (setup.py) ... done
Requirement already satisfied: xarray in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo==0.0.2) (2022.3.0)
Requirement already satisfied: julia in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo==0.0.2) (0.5.7)
Requirement already satisfied: tqdm in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo==0.0.2) (4.64.0)
Requirement already satisfied: numcodecs>=0.10.0 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo==0.0.2) (0.10.1.dev1)
Requirement already satisfied: typing-extensions>=3.7.4 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from numcodecs>=0.10.0->xbitinfo==0.0.2) (4.3.0)
Requirement already satisfied: numpy>=1.7 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from numcodecs>=0.10.0->xbitinfo==0.0.2) (1.23.0)
Requirement already satisfied: pandas>=1.1 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xarray->xbitinfo==0.0.2) (1.4.3)
Requirement already satisfied: packaging>=20.0 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xarray->xbitinfo==0.0.2) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from packaging>=20.0->xarray->xbitinfo==0.0.2) (3.0.9)
Requirement already satisfied: python-dateutil>=2.8.1 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from pandas>=1.1->xarray->xbitinfo==0.0.2) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from pandas>=1.1->xarray->xbitinfo==0.0.2) (2022.1)
Requirement already satisfied: six>=1.5 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1.1->xarray->xbitinfo==0.0.2) (1.16.0)
Building wheels for collected packages: xbitinfo
  Building wheel for xbitinfo (setup.py) ... done
  Created wheel for xbitinfo: filename=xbitinfo-0.0.2-py2.py3-none-any.whl size=20693 sha256=a17f07f5cc1f567d8898bcb014e450c4bda9d452a2e2224d14c35fa07c7b5c3b
  Stored in directory: /private/var/folders/fj/g0x4n_f15tb6zfwjhzc8gvzr0000gn/T/pip-ephem-wheel-cache-hy85njh8/wheels/7f/73/1d/68290b3559f4c7c26915da78a7bc69f3258dce9a13643de2bd
Successfully built xbitinfo
Installing collected packages: xbitinfo
Successfully installed xbitinfo-0.0.2

and

% pip install xbitinfo    
Collecting xbitinfo
  Downloading xbitinfo-0.0.2-py2.py3-none-any.whl (20 kB)
Requirement already satisfied: numcodecs>=0.10.0 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo) (0.10.1.dev1)
Requirement already satisfied: julia in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo) (0.5.7)
Requirement already satisfied: tqdm in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo) (4.64.0)
Requirement already satisfied: xarray in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xbitinfo) (2022.3.0)
Requirement already satisfied: typing-extensions>=3.7.4 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from numcodecs>=0.10.0->xbitinfo) (4.3.0)
Requirement already satisfied: numpy>=1.7 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from numcodecs>=0.10.0->xbitinfo) (1.23.0)
Requirement already satisfied: pandas>=1.1 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xarray->xbitinfo) (1.4.3)
Requirement already satisfied: packaging>=20.0 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from xarray->xbitinfo) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from packaging>=20.0->xarray->xbitinfo) (3.0.9)
Requirement already satisfied: python-dateutil>=2.8.1 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from pandas>=1.1->xarray->xbitinfo) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from pandas>=1.1->xarray->xbitinfo) (2022.1)
Requirement already satisfied: six>=1.5 in /Users/haukeschulz/mambaforge/envs/114/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas>=1.1->xarray->xbitinfo) (1.16.0)
Installing collected packages: xbitinfo
Successfully installed xbitinfo-0.0.2
aaronspring commented 2 years ago

Building wheels for collected packages: xbitinfo is missing somehow

observingClouds commented 2 years ago

Well, I think the wheels are those that we create beforehand and upload to pypi. I think we need to check the wheel building process again. The issue might be that PostInstallCommand https://github.com/observingClouds/xbitinfo/blob/4d5563e3416d61b26589abcdd32a08dd72fd361b/setup.py#L22-L27 is not executed when installing from Pypi.

@ocefpaf do you have an idea how to fix this?

ocefpaf commented 2 years ago

I'm actually looking into the best practices for shipping/installing julia packages when wrapping them with Python. So far, all I can say, is that calling os.system is a no-no :-/

In conda-forge we can easily ship a julia interpreter and make that a dependency. In PyPI I could not find any guidance yet.

PS: It would be nice if the underlying code was re-written in a more "wrapping friendly" language, like C/C++/Rust. Julia is easy to wrap but hard to ship b/c of the numerous patches they apply to llvm.

aaronspring commented 2 years ago

So a conda release could maybe work? Let’s try

observingClouds commented 2 years ago

I was actually thinking the same today! It's the best option we have currently.

aaronspring commented 2 years ago

Also to have it just installable in an easier way

ocefpaf commented 2 years ago

Just want to xref. this discussion here: https://github.com/conda-forge/pysr-feedstock/issues/38 Things are in flux but we can use that solution to make this package available on conda-forge at the moment.

observingClouds commented 2 years ago

Thanks to @rsignell-usgs we make great progress on the conda package: https://github.com/conda-forge/staged-recipes/pull/20116

rsignell-usgs commented 2 years ago

And @ocefpaf said he would help us with the Julia issues! Yay! Probably some time next week...

rsignell-usgs commented 2 years ago

Just a note here that you can now install xbitinfo on Linux using conda-forge:

conda install -c conda-forge xbitinfo

We installed it on the ESIP Qhub JupyterHub Deployment (on Kubernetes w/ Dask Gateway) and it's working great!

rsignell-usgs commented 2 years ago

The conda-forge recipe uses a patch -- @observingClouds would you be interested in making the changes in the repo here so we don't need that?

Did I get that right @ocefpaf ?

ocefpaf commented 2 years ago

Did I get that right @ocefpaf ?

Yep. The patch removed the post install command, which can be troublesome, and uses a install function that runs at import time to install and activate the Julia environment for xbitinfo. It is slow the first time you import it but it should skip the install steps the second time and perform only the activation.

In theory that works with both conda and pip. Should I send a PR to test that here?

observingClouds commented 2 years ago

Thank you so much guys! This is amazing! I'm happy to drop the pre/post-install commands if the suggested method works for both pip and conda.

@ocefpaf please do a PR, you guys should get the credit for this!

observingClouds commented 2 years ago

Closing as this should be fixed by #132

aaronspring commented 2 years ago

I pip install xbitinfo and julia isnt installed:

If you have installed Julia, make sure Julia executable is in the system PATH. Alternatively, specify file path to the Julia executable using julia keyword argument.

If you have not installed Julia, download Julia from https://julialang.org/downloads/ and install it.

ocefpaf commented 2 years ago

That is expected but we need a better error message to tell users to have a julia executable int he path. The useful bit of information (no pun intended) is lost in the traceback.

I'm not sure installing julia automatically is recommend. However, if there is a safe way to do that in all platforms we could try it.

aaronspring commented 2 years ago

pyjulia in the requirements is not enough. This was this previous solution

ocefpaf commented 2 years ago

pyjulia in the requirements is not enough. This was this previous solution

Interesting. I had no idea pyjulia would install julia when asked to do it and I may have disabled it then. I'll investigate this. However, my packager bias tells me that installing julia automatically is more trouble then it is worth. Again, I don't have experience with that kind of automatic installation, maybe it is fine nowadays and I'm outdated.

Note that their docs seems to tell users to handle the julia installation manually. See https://pyjulia.readthedocs.io/en/latest/installation.html#step-1-install-julia

PS: I tried and I'm not able to make it work with just the install call:

❯ pip install julia
Collecting julia
  Downloading julia-0.5.7-py2.py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.6/67.6 kB 1.9 MB/s eta 0:00:00
Installing collected packages: julia
Successfully installed julia-0.5.7
(TMP) 
~ via 🐍 v3.10.6 via 🅒 TMP 
❯ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:36:39) [GCC 10.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import install
>>> install()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/filipe/micromamba/envs/TMP/lib/python3.10/site-packages/julia/tools.py", line 78, in install
    raise JuliaNotFound(julia, kwargname="julia")
julia.core.JuliaNotFound: Julia executable `julia` cannot be found.

If you have installed Julia, make sure Julia executable is in the
system PATH.  Alternatively, specify file path to the Julia executable
using `julia` keyword argument.

If you have not installed Julia, download Julia from
https://julialang.org/downloads/ and install it.

I'm probably missing some detail here. I'll check the old code and get back when I find it.

observingClouds commented 2 years ago

I think we should just mention in the documentation/readme that the installation via conda is preferred and that one need to install Julia manually when using pip.

aaronspring commented 2 years ago

ok. lets close then