search

ray-project / ray_lightning

Pytorch Lightning Distributed Accelerators using Ray
Apache License 2.0
211 stars 34 forks source link

horovod installation issue #165

Open JiahaoYao opened 2 years ago

JiahaoYao commented 2 years ago

Using the flag to install horovod but met with the following issues. 

(tensorflow2_p38) ubuntu@ip-10-0-2-36:~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/ray_lightning$ HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITH_TORCH=1 HOROVOD_WITH_GLOO=1 pip install --no-cache-dir horovod[tensorflow]  horovod[ray] horovod[torch]
Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Collecting horovod[tensorflow]
  Downloading horovod-0.25.0.tar.gz (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 45.0 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: cloudpickle in /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages (from horovod[tensorflow]) (2.0.0)
Requirement already satisfied: psutil in /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages (from horovod[tensorflow]) (5.8.0)
Requirement already satisfied: pyyaml in /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages (from horovod[tensorflow]) (6.0)
Requirement already satisfied: cffi>=1.4.0 in /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages (from horovod[tensorflow]) (1.15.0)
Collecting tensorflow
  Downloading tensorflow-2.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 511.7/511.7 MB 191.4 MB/s eta 0:00:00

      creating build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      running build_ext
      Running CMake in build/temp.linux-x86_64-3.8/RelWithDebInfo:
      cmake /tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/lib.linux-x86_64-3.8 -DPYTHON_EXECUTABLE:FILEPATH=/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python
      cmake --build . --config RelWithDebInfo -- -j8 VERBOSE=1
      -- Could not find CCache. Consider installing CCache to speed up compilation.
      -- The CXX compiler identification is GNU 9.3.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/x86_64-conda-linux-gnu-c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build architecture flags: -mf16c -mavx -mfma
      -- Using command /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python
      -- Found MPI_CXX: /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/libmpi.so (found version "3.1")
      -- Found MPI: TRUE (found version "3.1")
      CMake Error at /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:724 (message):
        Compiling the CUDA compiler identification source file
        "CMakeCUDACompilerId.cu" failed.

        Compiler: /usr/local/cuda-11.2/bin/nvcc

        Build flags:

        Id flags:
        --keep;--keep-dir;tmp;-ccbin=/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/x86_64-conda-linux-gnu-c++
        -v

        The output was:

        1

        #$ _NVVM_BRANCH_=nvvm

        #$ _SPACE_=

        #$ _CUDART_=cudart

        #$ _HERE_=/usr/local/cuda-11.2/bin

        #$ _THERE_=/usr/local/cuda-11.2/bin

        #$ _TARGET_SIZE_=

        #$ _TARGET_DIR_=

        #$ _TARGET_DIR_=targets/x86_64-linux

        #$ TOP=/usr/local/cuda-11.2/bin/..

        #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda-11.2/bin/../nvvm/libdevice

        #$
        LD_LIBRARY_PATH=/usr/local/cuda-11.2/bin/../lib:/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/extras/CUPTI/lib64:/usr/local/cuda-11.2/lib:/usr/local/cuda-11.2/efa/lib:/opt/amazon/efa/lib:/opt/amazon/efa/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/lib:/usr/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/lib:/usr/lib::/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow

        #$
        PATH=/usr/local/cuda-11.2/bin/../nvvm/bin:/usr/local/cuda-11.2/bin:/usr/local/cuda-11.2/bin:/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin:/home/ubuntu/anaconda3/condabin:/home/ubuntu/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/bin/remote-cli:/home/ubuntu/anaconda3/condabin:/home/ubuntu/.dl_binaries/bin:/opt/aws/neuron/bin:/opt/amazon/openmpi/bin:/opt/amazon/efa/bin:/home/ubuntu/anaconda3/condabin:/home/ubuntu/.dl_binaries/bin:/opt/aws/neuron/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

        #$ INCLUDES="-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"

        #$ LIBRARIES=
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib/stubs"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib"

        #$ CUDAFE_FLAGS=

        #$ PTXAS_FLAGS=

        #$ rm tmp/a_dlink.reg.c

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -D__CUDA_ARCH__=520 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__
        -D__NVCC__ "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"
        -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2
        -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11
        -D__CUDA_API_VER_MINOR__=2 -include "cuda_runtime.h" -m64
        "CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.cpp1.ii"

        #$ cicc --c++14 --gnu_version=90300 --orig_src_file_name
        "CMakeCUDACompilerId.cu" --allow_managed -arch compute_52 -m64
        --no-version-ident -ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1
        --include_file_name "CMakeCUDACompilerId.fatbin.c" -tused
        --gen_module_id_file --module_id_file_name
        "tmp/CMakeCUDACompilerId.module_id" --gen_c_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.c" --stub_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.stub.c" --gen_device_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.gpu" "tmp/CMakeCUDACompilerId.cpp1.ii" -o
        "tmp/CMakeCUDACompilerId.ptx"

        #$ ptxas -arch=sm_52 -m64 "tmp/CMakeCUDACompilerId.ptx" -o
        "tmp/CMakeCUDACompilerId.sm_52.cubin"

        #$ fatbinary --create="tmp/CMakeCUDACompilerId.fatbin" -64
        --cicc-cmdline="-ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 "
        "--image3=kind=elf,sm=52,file=tmp/CMakeCUDACompilerId.sm_52.cubin"
        "--image3=kind=ptx,sm=52,file=tmp/CMakeCUDACompilerId.ptx"
        --embedded-fatbin="tmp/CMakeCUDACompilerId.fatbin.c"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -E -x c++ -D__CUDACC__ -D__NVCC__
        "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"
        -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2
        -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11
        -D__CUDA_API_VER_MINOR__=2 -include "cuda_runtime.h" -m64
        "CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.cpp4.ii"

        #$ cudafe++ --c++14 --gnu_version=90300 --orig_src_file_name
        "CMakeCUDACompilerId.cu" --allow_managed --m64 --parse_templates
        --gen_c_file_name "tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
        "CMakeCUDACompilerId.cudafe1.stub.c" --module_id_file_name
        "tmp/CMakeCUDACompilerId.module_id" "tmp/CMakeCUDACompilerId.cpp4.ii"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -D__CUDA_ARCH__=520 -c -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS
        "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include" -m64
        "tmp/CMakeCUDACompilerId.cudafe1.cpp" -o "tmp/CMakeCUDACompilerId.o"

        #$ nvlink --arch=sm_52 --register-link-binaries="tmp/a_dlink.reg.c" -m64
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib/stubs"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib" -cpu-arch=X86_64
        "tmp/CMakeCUDACompilerId.o" -lcudadevrt -o "tmp/a_dlink.sm_52.cubin"

        #$ fatbinary --create="tmp/a_dlink.fatbin" -64 --cicc-cmdline="-ftz=0
        -prec_div=1 -prec_sqrt=1 -fmad=1 " -link
        "--image3=kind=elf,sm=52,file=tmp/a_dlink.sm_52.cubin"
        --embedded-fatbin="tmp/a_dlink.fatbin.c"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -c -x c++ -DFATBINFILE="\"tmp/a_dlink.fatbin.c\""
        -DREGISTERLINKBINARYFILE="\"tmp/a_dlink.reg.c\"" -I.
        -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION=
        -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__
        "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"
        -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2
        -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11
        -D__CUDA_API_VER_MINOR__=2 -m64 "/usr/local/cuda-11.2/bin/crt/link.stub" -o
        "tmp/a_dlink.o"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -m64 -Wl,--start-group "tmp/a_dlink.o" "tmp/CMakeCUDACompilerId.o"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib/stubs"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib" -lcudadevrt
        -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -o "a.out"

        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld:
        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libstdc++.so:
        undefined reference to `aligned_alloc@GLIBC_2.16'

        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld:
        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libstdc++.so:
        undefined reference to `clock_gettime@GLIBC_2.17'

        collect2: error: ld returned 1 exit status

        # --error 0x1 --

      Call Stack (most recent call first):
        /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
        /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
        /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
        CMakeLists.txt:176 (enable_language)

      -- Configuring incomplete, errors occurred!
      See also "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/temp.linux-x86_64-3.8/RelWithDebInfo/CMakeFiles/CMakeOutput.log".
      See also "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/temp.linux-x86_64-3.8/RelWithDebInfo/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/setup.py", line 210, in <module>
          setup(name='horovod',
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
          _build_ext.build_ext.run(self)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/command/build_ext.py", line 340, in run
          self.build_extensions()
        File "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/setup.py", line 144, in build_extensions
          subprocess.check_call(command, cwd=cmake_build_dir)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/lib.linux-x86_64-3.8', '-DPYTHON_EXECUTABLE:FILEPATH=/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: hiredis, horovod, aioredis, tensorflow
  Running setup.py install for horovod ... error
  error: subprocess-exited-with-error

  × Running setup.py install for horovod did not run successfully.
  │ exit code: 1
  ╰─> [406 lines of output]
      running install
      /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.8
      creating build/lib.linux-x86_64-3.8/horovod
      copying horovod/__init__.py -> build/lib.linux-x86_64-3.8/horovod
      creating build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/strategy.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/worker.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/utils.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/ray_logger.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/elastic_v2.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/runner.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/__init__.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/elastic.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/adapter.py -> build/lib.linux-x86_64-3.8/horovod/ray
      copying horovod/ray/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/ray
      creating build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/launch.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/mpi_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/js_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/run_task.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/gloo_run.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/task_fn.py -> build/lib.linux-x86_64-3.8/horovod/runner
      copying horovod/runner/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner
      creating build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/process_sets.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/exceptions.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/util.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/basics.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/common
      copying horovod/common/elastic.py -> build/lib.linux-x86_64-3.8/horovod/common
      creating build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/mpi_run.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/conf.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/gloo_run.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/runner.py -> build/lib.linux-x86_64-3.8/horovod/spark
      copying horovod/spark/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark
      creating build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/functions.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/compression.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/__init__.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/mxnet
      creating build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/_keras
      creating build/lib.linux-x86_64-3.8/horovod/data
      copying horovod/data/data_loader_base.py -> build/lib.linux-x86_64-3.8/horovod/data
      copying horovod/data/__init__.py -> build/lib.linux-x86_64-3.8/horovod/data
      creating build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/functions.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/optimizer.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/compression.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/sync_batch_norm.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch
      copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/torch
      creating build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/keras
      copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/keras
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/functions.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation_eager.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/sync_batch_norm.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow
      creating build/lib.linux-x86_64-3.8/horovod/runner/driver
      copying horovod/runner/driver/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/driver
      copying horovod/runner/driver/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/driver
      creating build/lib.linux-x86_64-3.8/horovod/runner/common
      copying horovod/runner/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common
      creating build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/driver.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/worker.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/constants.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/registration.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/rendezvous.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/discovery.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      copying horovod/runner/elastic/settings.py -> build/lib.linux-x86_64-3.8/horovod/runner/elastic
      creating build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/threads.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/cache.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/remote.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/streams.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/network.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/lsf.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      copying horovod/runner/util/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/util
      creating build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/http_server.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      copying horovod/runner/http/http_client.py -> build/lib.linux-x86_64-3.8/horovod/runner/http
      creating build/lib.linux-x86_64-3.8/horovod/runner/task
      copying horovod/runner/task/task_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/task
      copying horovod/runner/task/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/task
      creating build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/task_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      copying horovod/runner/common/service/compute_service.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/service
      creating build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/network.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/tiny_shell_exec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/secret.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/config_parser.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/host_hash.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/timeout.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/__init__.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/env.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/codec.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/settings.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      copying horovod/runner/common/util/hosts.py -> build/lib.linux-x86_64-3.8/horovod/runner/common/util
      creating build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/rsh.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/host_discovery.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/rendezvous.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-3.8/horovod/spark/driver
      creating build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/params.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/cache.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/backend.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/constants.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/store.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/serialization.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/_namedtuple_fix.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      copying horovod/spark/common/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/common
      creating build/lib.linux-x86_64-3.8/horovod/spark/lightning
      copying horovod/spark/lightning/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/lightning
      copying horovod/spark/lightning/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/lightning
      copying horovod/spark/lightning/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/lightning
      copying horovod/spark/lightning/legacy.py -> build/lib.linux-x86_64-3.8/horovod/spark/lightning
      copying horovod/spark/lightning/datamodule.py -> build/lib.linux-x86_64-3.8/horovod/spark/lightning
      copying horovod/spark/lightning/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/lightning
      creating build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      copying horovod/spark/torch/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/torch
      creating build/lib.linux-x86_64-3.8/horovod/spark/data_loaders
      copying horovod/spark/data_loaders/pytorch_data_loaders.py -> build/lib.linux-x86_64-3.8/horovod/spark/data_loaders
      copying horovod/spark/data_loaders/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/data_loaders
      creating build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/estimator.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/remote.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/optimizer.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/tensorflow.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/bare.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/util.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      copying horovod/spark/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/keras
      creating build/lib.linux-x86_64-3.8/horovod/spark/tensorflow
      copying horovod/spark/tensorflow/compute_worker.py -> build/lib.linux-x86_64-3.8/horovod/spark/tensorflow
      copying horovod/spark/tensorflow/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/tensorflow
      creating build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/task_info.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/gloo_exec_fn.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      copying horovod/spark/task/__init__.py -> build/lib.linux-x86_64-3.8/horovod/spark/task
      creating build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/state.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/sampler.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      copying horovod/torch/elastic/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/elastic
      creating build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib
      copying horovod/torch/mpi_lib/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib
      creating build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib_impl
      copying horovod/torch/mpi_lib_impl/__init__.py -> build/lib.linux-x86_64-3.8/horovod/torch/mpi_lib_impl
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow/data
      copying horovod/tensorflow/data/compute_worker.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/data
      copying horovod/tensorflow/data/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/data
      copying horovod/tensorflow/data/compute_service.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/data
      creating build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/elastic.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-3.8/horovod/tensorflow/keras
      running build_ext
      Running CMake in build/temp.linux-x86_64-3.8/RelWithDebInfo:
      cmake /tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/lib.linux-x86_64-3.8 -DPYTHON_EXECUTABLE:FILEPATH=/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python
      cmake --build . --config RelWithDebInfo -- -j8 VERBOSE=1
      -- Could not find CCache. Consider installing CCache to speed up compilation.
      -- The CXX compiler identification is GNU 9.3.0
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/x86_64-conda-linux-gnu-c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Build architecture flags: -mf16c -mavx -mfma
      -- Using command /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python
      -- Found MPI_CXX: /home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/libmpi.so (found version "3.1")
      -- Found MPI: TRUE (found version "3.1")
      CMake Error at /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:724 (message):
        Compiling the CUDA compiler identification source file
        "CMakeCUDACompilerId.cu" failed.

        Compiler: /usr/local/cuda-11.2/bin/nvcc

        Build flags:

        Id flags:
        --keep;--keep-dir;tmp;-ccbin=/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/x86_64-conda-linux-gnu-c++
        -v

        The output was:

        1

        #$ _NVVM_BRANCH_=nvvm

        #$ _SPACE_=

        #$ _CUDART_=cudart

        #$ _HERE_=/usr/local/cuda-11.2/bin

        #$ _THERE_=/usr/local/cuda-11.2/bin

        #$ _TARGET_SIZE_=

        #$ _TARGET_DIR_=

        #$ _TARGET_DIR_=targets/x86_64-linux

        #$ TOP=/usr/local/cuda-11.2/bin/..

        #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda-11.2/bin/../nvvm/libdevice

        #$
        LD_LIBRARY_PATH=/usr/local/cuda-11.2/bin/../lib:/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/extras/CUPTI/lib64:/usr/local/cuda-11.2/lib:/usr/local/cuda-11.2/efa/lib:/opt/amazon/efa/lib:/opt/amazon/efa/lib64:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/lib:/usr/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/usr/local/lib:/usr/lib::/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow

        #$
        PATH=/usr/local/cuda-11.2/bin/../nvvm/bin:/usr/local/cuda-11.2/bin:/usr/local/cuda-11.2/bin:/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin:/home/ubuntu/anaconda3/condabin:/home/ubuntu/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/bin/remote-cli:/home/ubuntu/anaconda3/condabin:/home/ubuntu/.dl_binaries/bin:/opt/aws/neuron/bin:/opt/amazon/openmpi/bin:/opt/amazon/efa/bin:/home/ubuntu/anaconda3/condabin:/home/ubuntu/.dl_binaries/bin:/opt/aws/neuron/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

        #$ INCLUDES="-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"

        #$ LIBRARIES=
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib/stubs"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib"

        #$ CUDAFE_FLAGS=

        #$ PTXAS_FLAGS=

        #$ rm tmp/a_dlink.reg.c

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -D__CUDA_ARCH__=520 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__
        -D__NVCC__ "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"
        -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2
        -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11
        -D__CUDA_API_VER_MINOR__=2 -include "cuda_runtime.h" -m64
        "CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.cpp1.ii"

        #$ cicc --c++14 --gnu_version=90300 --orig_src_file_name
        "CMakeCUDACompilerId.cu" --allow_managed -arch compute_52 -m64
        --no-version-ident -ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1
        --include_file_name "CMakeCUDACompilerId.fatbin.c" -tused
        --gen_module_id_file --module_id_file_name
        "tmp/CMakeCUDACompilerId.module_id" --gen_c_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.c" --stub_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.stub.c" --gen_device_file_name
        "tmp/CMakeCUDACompilerId.cudafe1.gpu" "tmp/CMakeCUDACompilerId.cpp1.ii" -o
        "tmp/CMakeCUDACompilerId.ptx"

        #$ ptxas -arch=sm_52 -m64 "tmp/CMakeCUDACompilerId.ptx" -o
        "tmp/CMakeCUDACompilerId.sm_52.cubin"

        #$ fatbinary --create="tmp/CMakeCUDACompilerId.fatbin" -64
        --cicc-cmdline="-ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 "
        "--image3=kind=elf,sm=52,file=tmp/CMakeCUDACompilerId.sm_52.cubin"
        "--image3=kind=ptx,sm=52,file=tmp/CMakeCUDACompilerId.ptx"
        --embedded-fatbin="tmp/CMakeCUDACompilerId.fatbin.c"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -E -x c++ -D__CUDACC__ -D__NVCC__
        "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"
        -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2
        -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11
        -D__CUDA_API_VER_MINOR__=2 -include "cuda_runtime.h" -m64
        "CMakeCUDACompilerId.cu" -o "tmp/CMakeCUDACompilerId.cpp4.ii"

        #$ cudafe++ --c++14 --gnu_version=90300 --orig_src_file_name
        "CMakeCUDACompilerId.cu" --allow_managed --m64 --parse_templates
        --gen_c_file_name "tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
        "CMakeCUDACompilerId.cudafe1.stub.c" --module_id_file_name
        "tmp/CMakeCUDACompilerId.module_id" "tmp/CMakeCUDACompilerId.cpp4.ii"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -D__CUDA_ARCH__=520 -c -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS
        "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include" -m64
        "tmp/CMakeCUDACompilerId.cudafe1.cpp" -o "tmp/CMakeCUDACompilerId.o"

        #$ nvlink --arch=sm_52 --register-link-binaries="tmp/a_dlink.reg.c" -m64
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib/stubs"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib" -cpu-arch=X86_64
        "tmp/CMakeCUDACompilerId.o" -lcudadevrt -o "tmp/a_dlink.sm_52.cubin"

        #$ fatbinary --create="tmp/a_dlink.fatbin" -64 --cicc-cmdline="-ftz=0
        -prec_div=1 -prec_sqrt=1 -fmad=1 " -link
        "--image3=kind=elf,sm=52,file=tmp/a_dlink.sm_52.cubin"
        --embedded-fatbin="tmp/a_dlink.fatbin.c"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -c -x c++ -DFATBINFILE="\"tmp/a_dlink.fatbin.c\""
        -DREGISTERLINKBINARYFILE="\"tmp/a_dlink.reg.c\"" -I.
        -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION=
        -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__
        "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include"
        -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2
        -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11
        -D__CUDA_API_VER_MINOR__=2 -m64 "/usr/local/cuda-11.2/bin/crt/link.stub" -o
        "tmp/a_dlink.o"

        #$
        "/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin"/x86_64-conda-linux-gnu-c++
        -m64 -Wl,--start-group "tmp/a_dlink.o" "tmp/CMakeCUDACompilerId.o"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib/stubs"
        "-L/usr/local/cuda-11.2/bin/../targets/x86_64-linux/lib" -lcudadevrt
        -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -o "a.out"

        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld:
        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libstdc++.so:
        undefined reference to `aligned_alloc@GLIBC_2.16'

        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld:
        /home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libstdc++.so:
        undefined reference to `clock_gettime@GLIBC_2.17'

        collect2: error: ld returned 1 exit status

        # --error 0x1 --

      Call Stack (most recent call first):
        /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
        /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
        /usr/local/lib/python3.6/dist-packages/cmake/data/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
        CMakeLists.txt:176 (enable_language)

      -- Configuring incomplete, errors occurred!
      See also "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/temp.linux-x86_64-3.8/RelWithDebInfo/CMakeFiles/CMakeOutput.log".
      See also "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/temp.linux-x86_64-3.8/RelWithDebInfo/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/setup.py", line 210, in <module>
          setup(name='horovod',
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/setuptools/command/install.py", line 68, in run
          return orig.install.run(self)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/command/install.py", line 545, in run
          self.run_command('build')
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
          _build_ext.run(self)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
          _build_ext.build_ext.run(self)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/distutils/command/build_ext.py", line 340, in run
          self.build_extensions()
        File "/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/setup.py", line 144, in build_extensions
          subprocess.check_call(command, cwd=cmake_build_dir)
        File "/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELWITHDEBINFO=/tmp/pip-install-az9tmh2e/horovod_e2db9da803c84a82a5916756412363a2/build/lib.linux-x86_64-3.8', '-DPYTHON_EXECUTABLE:FILEPATH=/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> horovod

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/home/ubuntu/anaconda3/envs/tensorflow2_p38/bin/python -m pip install --upgrade pip' command.
amogkam commented 2 years ago

I was getting this error with ray installation from master as well.

Is there anything about this on the issues in the horovod repo?