ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.54k stars 5.69k forks source link

Issue when installing ray on the server #4751

Closed stellaxu closed 3 years ago

stellaxu commented 5 years ago

I have tried the following two ways to install ray: i) git clone https://github.com/ray-project/ray.git

Install Bazel.

ray/ci/travis/install-bazel.sh

cd ray/python pip install -e . --verbose # Add --user if you see a permission denied error.

and ii) pip install git+https://github.com/ray-project/ray.git#subdirectory=python

However, both of the installation methods make me encounter the same issues like the followings and the installation fails:

Would you please help me to figure it out?

Thanks a lot.

robertnishihara commented 5 years ago

I suspect that is not the real error (since it just says "warning"). Can you include the full output from installation?

Also, if you want to build from source, then I recommend first cloning the repository and then installing from the local copy.

If you don't plan on modifying Ray, then I'd suggest not building from source and instead using the wheels that we publish, e.g., pip install -U ray. If you want the wheels for the latest commits, follow the instructions at https://ray.readthedocs.io/en/latest/installation.html#trying-snapshots-from-master.

stellaxu commented 5 years ago

Hi, Robert: Since I need to add algorithms to RLlib, perhaps I have to build from source.

The full output (starting from the point a little bit before the error occurs) is the following: Installing collected packages: numpy, six, pyarrow Successfully installed numpy-1.16.3 pyarrow-0.12.0.RAY six-1.12.0

stellaxu commented 5 years ago

In addition, what do you mean by installing from the local copy? thxs

tmorgan4 commented 5 years ago

FYI, you can install ray via pip and then install just the tune and rllib libraries in development mode. Unfortunately it results in a very fragile install that will delete your tune and rllib folders if you upgrade ray without removing the aliases first. Details are found here: https://ray.readthedocs.io/en/latest/rllib-dev.html#development-install

stellaxu commented 5 years ago

Hi,currently there is another bug for installation. Would you please help me to figure it out? Thxs a lot. The debug output is as follows:

+ /home/eecs/xuweinan/.bazel/bin/bazel build //:ray_pkg --verbose_failures  
WARNING: Output base '/home/eecs/xuweinan/.cache/bazel/_bazel_xuweinan/c6c4c

0c6c37ad0fdc6b363526c0a5807' is on NFS. This may lead to surprising failures and undetermined behavior.
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 4ccf97d9-896c-4694-9b10-9326c6267ae9
Loading:
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
currently loading:
Analyzing: target //:ray_pkg (1 packages loaded, 0 targets configured)
Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured)
Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured) Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured) Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured) Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured) Analyzing: target //:ray_pkg (3 packages loaded, 16 targets configured) ERROR: /work/xuweinan/ray/BUILD.bazel:611:1:no such package '@plasma//': Tr aceback (most recent call last):
File "/home/eecs/xuweinan/.cache/bazel/_bazel_xuweinan/c6c4c0c6c37ad0fdc 6b363526c0a5807/external/bazel_tools/tools/build_defs/repo/git.bzl", line 163
_clone_or_update(ctx)
File "/home/eecs/xuweinan/.cache/bazel/_bazel_xuweinan/c6c4c0c6c37ad0fdc 6b363526c0a5807/external/bazel_tools/tools/build_defs/repo/git.bzl", line 73, in _clone_or_update
fail(("error cloning %s:\n%s" % (ctx....)))
error cloning plasma:
Timed out; also encountered an error while attempting to retrieve output and referenced by '//:ray_pkg'
ERROR: Analysis of target '//:ray_pkg' failed; build aborted: no such packag e '@plasma//': Traceback (most recent call last): File "/home/eecs/xuweinan/.cache/bazel/_bazel_xuweinan/c6c4c0c6c37ad0fdc 6b363526c0a5807/external/bazel_tools/tools/build_defs/repo/git.bzl", line 163 _clone_or_update(ctx) File "/home/eecs/xuweinan/.cache/bazel/_bazel_xuweinan/c6c4c0c6c37ad0fdc 6b363526c0a5807/external/bazel_tools/tools/build_defs/repo/git.bzl", line 73, in _clone_or_update fail(("error cloning %s:\n%s" % (ctx....))) error cloning plasma: Timed out; also encountered an error while attempting to retrieve output INFO: Elapsed time: 885.971s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (3 packages loaded, 16 targets c onfigured) FAILED: Build did NOT complete successfully (3 packages loaded, 16 targets c onfigured) Traceback (most recent call last): File "", line 1, in File "/work/xuweinan/ray/python/setup.py", line 183, in license="Apache 2.0") File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/setuptools/in it.py", line 145, in setup return distutils.core.setup(attrs) File "/work/xuweinan/anaconda3/lib/python3.6/distutils/core.py", line 148, in setup dist.run_commands() File "/work/xuweinan/anaconda3/lib/python3.6/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/work/xuweinan/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/setuptools/comm and/develop.py", line 38, in run
self.install_for_development()
File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/setuptools/comm and/develop.py", line 140, in install_for_development
self.run_command('build_ext')
File "/work/xuweinan/anaconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/work/xuweinan/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/work/xuweinan/ray/python/setup.py", line 74, in run subprocess.check_call(command) File "/work/xuweinan/anaconda3/lib/python3.6/subprocess.py", line 311, in checkcall raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['../build.sh', '-p', '/work/xuweina n/anaconda3/bin/python']' returned non-zero exit status 1.
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-ueb1nia
'
ERROR: Command "/work/xuweinan/anaconda3/bin/python -c 'import setuptools, token ize;file='"'"'/work/xuweinan/ray/python/setup.py'"'"';f=getattr(tokenize, '" '"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"' );f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps" fa iled with error code 1 in /work/xuweinan/ray/python/
Exception information:
Traceback (most recent call last):
File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/pip/_internal/cli/b ase_command.py", line 178, in main
status = self.run(options, args)
File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/pip/_internal/comm$ nds/install.py", line 414, in run
use_user_site=options.use_user_site,
File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/pip/internal/req/ _init__.py", line 58, in install_given_reqs
kwargs File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/pip/_internal/req/r eq_install.py", line 768, in install_editable cwd=self.setup_py_dir, File "/work/xuweinan/anaconda3/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 776, in call_subprocess % (command_desc, proc.returncode, cwd)) pip._internal.exceptions.InstallationError: Command "/work/xuweinan/anaconda3/bin/python -c 'import setuptools, tokenize;file='"'"'/work/xuweinan/ray/python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps" failed with error code 1 in /work/xuweinan/ray/python/

robertnishihara commented 5 years ago

It seems like the error is related to cloning plasma... could there be any issue with the internet? Does the issue persist consistently?

stellaxu commented 5 years ago

Hi, Robert. Yes, this same issue happens every time when I installed on the atlas cluster, and is there any way to solve it? Thxs.

robertnishihara commented 5 years ago

What happens if you run git clone https://github.com/apache/arrow.git on that machine? Does it work?

I'm not really sure what the issue is unfortunately.

stellaxu commented 5 years ago

Hi, Robert. I run the git code, but the installation still fails due to the same reason "ERROR: /work/xuweinan/ray/BUILD.bazel:611:1:no such package '@plasma//': Tr aceback (most recent call last):". I think if this issue can be solved, then the installation on atlas would succeed.

developeralgo8888 commented 5 years ago

i am having similar issues installing on UBUNTU 18.04 due to opencensus issues . when i try to build ray from the sources . i am using [ bazel release 0.25.3 ] . I also did the same process as @robertnishihara , but all did have the same failure. i even cleared all the wheel cache but that didnt help

` Installing collected packages: six, numpy, pyarrow Successfully installed numpy-1.16.3 pyarrow-0.12.0.RAY six-1.12.0 WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/pyarrow-0.12.0.RAY.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/six.py already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/pycache already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/numpy already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/six-1.12.0.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/numpy-1.16.3.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/pyarrow already exists. Specify --upgrade to force replacement. WARNING: Target directory /developer/Downloads/REINFORCEMENT_LEARNING_2019/ray/python/ray/pyarrow_files/bin already exists. Specify --upgrade to force replacement.

stellaxu commented 5 years ago

Hi, all, I have solved this issue by setting: TEST_TMPDIR=/tmp/bazel before cloning the repository. The installation of Ray succeeds using the clone the repository and pip install -e. verbose method. Even when

Since by default, the bazel's output base is on NFS. Although it is a warning, it still make the installation fails.

Hope it works for both the potential users as well as the Ray developers.

stale[bot] commented 3 years ago

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

You can always ask for help on our discussion forum or Ray's public slack channel.

stale[bot] commented 3 years ago

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!