search

didi / athena

A release version for https://github.com/athena-team/athena
Apache License 2.0
125 stars 35 forks source link

install error: Running setup.py install for horovod ... error #31

Open LuxuryApple opened 4 years ago

LuxuryApple commented 4 years ago

Install Environment： VMware15.0, Ubuntu 18.04, python3.6.9

*Error:*** Running setup.py install for horovod ... error ERROR: Command errored out with exit status 1: command: /home/luxury/luxy/venv_athena/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s_f3vxnr/horovod/setup.py'"'"'; file='"'"'/tmp/pip-install-s_f3vxnr/horovod/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uelars4k/install-record.txt --single-version-externally-managed --compile --install-headers /home/luxury/luxy/venv_athena/include/site/python3.6/horovod cwd: /tmp/pip-install-s_f3vxnr/horovod/ Complete output (190 lines): running install running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/horovod copying horovod/init.py -> build/lib.linux-x86_64-3.6/horovod creating build/lib.linux-x86_64-3.6/horovod/keras copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-3.6/horovod/keras copying horovod/keras/init.py -> build/lib.linux-x86_64-3.6/horovod/keras creating build/lib.linux-x86_64-3.6/horovod/tensorflow copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-3.6/horovod/tensorflow copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-3.6/horovod/tensorflow copying horovod/tensorflow/init.py -> build/lib.linux-x86_64-3.6/horovod/tensorflow copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-3.6/horovod/tensorflow creating build/lib.linux-x86_64-3.6/horovod/run copying horovod/run/run_task.py -> build/lib.linux-x86_64-3.6/horovod/run copying horovod/run/gloo_run.py -> build/lib.linux-x86_64-3.6/horovod/run copying horovod/run/init.py -> build/lib.linux-x86_64-3.6/horovod/run copying horovod/run/run.py -> build/lib.linux-x86_64-3.6/horovod/run copying horovod/run/mpi_run.py -> build/lib.linux-x86_64-3.6/horovod/run copying horovod/run/task_fn.py -> build/lib.linux-x86_64-3.6/horovod/run creating build/lib.linux-x86_64-3.6/horovod/spark copying horovod/spark/init.py -> build/lib.linux-x86_64-3.6/horovod/spark creating build/lib.linux-x86_64-3.6/horovod/common copying horovod/common/util.py -> build/lib.linux-x86_64-3.6/horovod/common copying horovod/common/init.py -> build/lib.linux-x86_64-3.6/horovod/common copying horovod/common/basics.py -> build/lib.linux-x86_64-3.6/horovod/common creating build/lib.linux-x86_64-3.6/horovod/mxnet copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-3.6/horovod/mxnet copying horovod/mxnet/init.py -> build/lib.linux-x86_64-3.6/horovod/mxnet creating build/lib.linux-x86_64-3.6/horovod/_keras copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-3.6/horovod/_keras copying horovod/_keras/init.py -> build/lib.linux-x86_64-3.6/horovod/_keras creating build/lib.linux-x86_64-3.6/horovod/torch copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-3.6/horovod/torch copying horovod/torch/init.py -> build/lib.linux-x86_64-3.6/horovod/torch copying horovod/torch/compression.py -> build/lib.linux-x86_64-3.6/horovod/torch creating build/lib.linux-x86_64-3.6/horovod/tensorflow/keras copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-3.6/horovod/tensorflow/keras copying horovod/tensorflow/keras/init.py -> build/lib.linux-x86_64-3.6/horovod/tensorflow/keras creating build/lib.linux-x86_64-3.6/horovod/run/task copying horovod/run/task/task_service.py -> build/lib.linux-x86_64-3.6/horovod/run/task copying horovod/run/task/init.py -> build/lib.linux-x86_64-3.6/horovod/run/task creating build/lib.linux-x86_64-3.6/horovod/run/http copying horovod/run/http/http_client.py -> build/lib.linux-x86_64-3.6/horovod/run/http copying horovod/run/http/init.py -> build/lib.linux-x86_64-3.6/horovod/run/http copying horovod/run/http/http_server.py -> build/lib.linux-x86_64-3.6/horovod/run/http creating build/lib.linux-x86_64-3.6/horovod/run/common copying horovod/run/common/init.py -> build/lib.linux-x86_64-3.6/horovod/run/common creating build/lib.linux-x86_64-3.6/horovod/run/util copying horovod/run/util/network.py -> build/lib.linux-x86_64-3.6/horovod/run/util copying horovod/run/util/init.py -> build/lib.linux-x86_64-3.6/horovod/run/util copying horovod/run/util/cache.py -> build/lib.linux-x86_64-3.6/horovod/run/util copying horovod/run/util/threads.py -> build/lib.linux-x86_64-3.6/horovod/run/util creating build/lib.linux-x86_64-3.6/horovod/run/driver copying horovod/run/driver/driver_service.py -> build/lib.linux-x86_64-3.6/horovod/run/driver copying horovod/run/driver/init.py -> build/lib.linux-x86_64-3.6/horovod/run/driver creating build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/timeout.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/config_parser.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/secret.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/network.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/settings.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/codec.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/init.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/host_hash.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util copying horovod/run/common/util/env.py -> build/lib.linux-x86_64-3.6/horovod/run/common/util creating build/lib.linux-x86_64-3.6/horovod/run/common/service copying horovod/run/common/service/task_service.py -> build/lib.linux-x86_64-3.6/horovod/run/common/service copying horovod/run/common/service/driver_service.py -> build/lib.linux-x86_64-3.6/horovod/run/common/service copying horovod/run/common/service/init.py -> build/lib.linux-x86_64-3.6/horovod/run/common/service creating build/lib.linux-x86_64-3.6/horovod/spark/task copying horovod/spark/task/task_info.py -> build/lib.linux-x86_64-3.6/horovod/spark/task copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-3.6/horovod/spark/task copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-3.6/horovod/spark/task copying horovod/spark/task/init.py -> build/lib.linux-x86_64-3.6/horovod/spark/task creating build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/estimator.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/tensorflow.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/util.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/bare.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/init.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/remote.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras copying horovod/spark/keras/optimizer.py -> build/lib.linux-x86_64-3.6/horovod/spark/keras creating build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/estimator.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/util.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/store.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/_namedtuple_fix.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/constants.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/params.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/init.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/cache.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/serialization.py -> build/lib.linux-x86_64-3.6/horovod/spark/common copying horovod/spark/common/backend.py -> build/lib.linux-x86_64-3.6/horovod/spark/common creating build/lib.linux-x86_64-3.6/horovod/spark/driver copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-3.6/horovod/spark/driver copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-3.6/horovod/spark/driver copying horovod/spark/driver/init.py -> build/lib.linux-x86_64-3.6/horovod/spark/driver copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-3.6/horovod/spark/driver creating build/lib.linux-x86_64-3.6/horovod/spark/torch copying horovod/spark/torch/estimator.py -> build/lib.linux-x86_64-3.6/horovod/spark/torch copying horovod/spark/torch/util.py -> build/lib.linux-x86_64-3.6/horovod/spark/torch copying horovod/spark/torch/init.py -> build/lib.linux-x86_64-3.6/horovod/spark/torch copying horovod/spark/torch/remote.py -> build/lib.linux-x86_64-3.6/horovod/spark/torch creating build/lib.linux-x86_64-3.6/horovod/torch/mpi_lib copying horovod/torch/mpi_lib/init.py -> build/lib.linux-x86_64-3.6/horovod/torch/mpi_lib creating build/lib.linux-x86_64-3.6/horovod/torch/mpi_lib_impl copying horovod/torch/mpi_lib_impl/init.py -> build/lib.linux-x86_64-3.6/horovod/torch/mpi_lib_impl running build_ext x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -std=c++11 -fPIC -O2 -Wall -fassociative-math -ffast-math -ftree-vectorize -funsafe-math-optimizations -mf16c -mavx -mfma -I/home/luxury/luxy/venv_athena/include -I/usr/include/python3.6m -c build/temp.linux-x86_64-3.6/test_compile/test_cpp_flags.cc -o build/temp.linux-x86_64-3.6/test_compile/test_cpp_flags.o x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/test_compile/test_cpp_flags.o -o build/temp.linux-x86_64-3.6/test_compile/test_cpp_flags.so x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/luxury/luxy/venv_athena/include -I/usr/include/python3.6m -c build/temp.linux-x86_64-3.6/test_compile/test_link_flags.cc -o build/temp.linux-x86_64-3.6/test_compile/test_link_flags.o x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,--version-script=horovod.lds build/temp.linux-x86_64-3.6/test_compile/test_link_flags.o -o build/temp.linux-x86_64-3.6/test_compile/test_link_flags.so INFO: Cannot find CMake, will skip compiling Horovod with Gloo. Traceback (most recent call last): File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 341, in get_mpi_flags shlex.split(show_command), universal_newlines=True).strip() File "/usr/lib/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 423, in run with Popen(popenargs, kwargs) as process: File "/usr/lib/python3.6/subprocess.py", line 729, in init restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'mpicxx': 'mpicxx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 622, in get_common_options
    mpi_flags = get_mpi_flags()
  File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 354, in get_mpi_flags
    '%s' % (show_command, traceback.format_exc()))
distutils.errors.DistutilsPlatformError: mpicxx -show failed (see error below), is MPI in $PATH?
Note: If your version of MPI has a custom command to show compilation flags, please specify it with the HOROVOD_MPICXX_SHOW environment variable.

Traceback (most recent call last):
  File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 341, in get_mpi_flags
    shlex.split(show_command), universal_newlines=True).strip()
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'mpicxx': 'mpicxx'

INFO: Cannot find MPI compilation flags, will skip compiling with MPI.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 1566, in <module>
    scripts=['bin/horovodrun'])
  File "/home/luxury/luxy/venv_athena/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/luxury/luxy/venv_athena/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib/python3.6/distutils/command/install.py", line 589, in run
    self.run_command('build')
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/luxury/luxy/venv_athena/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/usr/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 1457, in build_extensions
    options = get_common_options(self)
  File "/tmp/pip-install-s_f3vxnr/horovod/setup.py", line 635, in get_common_options
    raise RuntimeError('One of Gloo or MPI are required for Horovod to run. Check the logs above for more info.')
RuntimeError: One of Gloo or MPI are required for Horovod to run. Check the logs above for more info.
----------------------------------------

ERROR: Command errored out with exit status 1: /home/luxury/luxy/venv_athena/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s_f3vxnr/horovod/setup.py'"'"'; file='"'"'/tmp/pip-install-s_f3vxnr/horovod/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uelars4k/install-record.txt --single-version-externally-managed --compile --install-headers /home/luxury/luxy/venv_athena/include/site/python3.6/horovod Check the logs for full command output.

How to solve it ?

Some-random commented 4 years ago

Some modules of this repo isn't up-to-date. Please raise issues regarding Athena at https://github.com/athena-team/athena.