search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.07k stars 5.79k forks source link

Ray 0.8.0.dev error on pyarrow 0.12.0 #4883

Closed developeralgo8888 closed 4 years ago

developeralgo8888 commented 5 years ago

System information

why is RAY giving me this Error on pyarrow , while pyarrow is installed correctly and works ok ????

why built from source ? ans: because as a quant i am constantly adding various agents/algorithms/envs to ray

ray & modin Versions are built from source:

ray 0.8.0.dev modin 0.6.0.dev pyarrow 0.12.0.RAY numpy 1.16.3

Checking if ray installed correctly:

(base) developer@DEVELOPER-AI:~# python Python 3.7.3 (default, Mar 27 2019, 22:11:17) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import numpy import six import modin import pyarrow import from pyarrow.lib import cpu_count, set_cpu_count

Checking if pyarrow, six, modin, numpy are installed correctly: --- i get an error for pyarrow when importing ray 0.8.0.dev

(base) developer@DEVELOPER-AI:~# python Python 3.7.3 (default, Mar 27 2019, 22:11:17) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.

import numpy import six import modin import ray Traceback (most recent call last): File "", line 1, in File "/developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/init.py", line 28, in import pyarrow # noqa: F401 File "/developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/pyarrow/init.py", line 47, in from pyarrow.lib import cpu_count, set_cpu_count ModuleNotFoundError: No module named 'pyarrow.lib'

Running ray test to check installation: --- i get an error for pyarrow when importing ray 0.8.0.dev

(base) developer@DEVELOPER-AI:~/Downloads/REINFORCEMENT_LEARNING_2019/ray# python -m pytest -v python/ray/tests/test_mini.py ImportError while loading conftest '/developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/tests/conftest.py'. python/ray/init.py:28: in import pyarrow # noqa: F401 python/ray/pyarrow_files/pyarrow/init.py:47: in from pyarrow.lib import cpu_count, set_cpu_count E ModuleNotFoundError: No module named 'pyarrow.lib'

Describe the problem

why is RAY giving me this Error on pyarrow , while pyarrow is installed correctly and works ok ????

Source code / logs

developeralgo8888 commented 5 years ago

I just found out that ray 0.8.0.dev collected packages (pyarrow, modin, numpy & six ) are never installed at all . When building ray 0.8.0.dev from source it shows that they are collected and successfully installed . i checked this by removing any of the following packages --pyarrow, numpy, modin, six . Then tried to see if they are actually installed during the build process. Unfortunately they are not installed at all. if i install them manually , they install successfully but then ray 0.8.0.dev doesn't work with pyarrow. The packages being collected are

-- pyarrow-0.12.0.RAY-cp37-cp37m-manylinux1_x86_64.whl -- modin-0.6.0.dev.0-py3-none-any.whl -- numpy-1.16.3-cp37-cp37m-manylinux1_x86_64.whl -- six-1.12.0-py2.py3-none-any.whl

i am using the following command to build and install as per documentation while in /ray/python source directory:

pip install -e . --verbose --upgrade

am i missing something during the build process ?????

pcmoritz commented 5 years ago

So Ray comes with its own version of pyarrow, which is registered by https://github.com/ray-project/ray/blob/master/python/ray/__init__.py#L15.

Could it be that the pyarrow installation failed when you were installing ray from source? Pyarrow is installed in https://github.com/ray-project/ray/blob/master/build.sh#L103 Maybe you can try rerunning that command and see if that fixes the problem.

pcmoritz commented 4 years ago

This is fixed now since we do not vendor pyarrow any more.