search

pytorch / pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration
https://pytorch.org
Other
84.5k stars 22.76k forks source link

get Error about install pytorch-mpi when doing environment configuration in a essay #63850

Closed YangWenhao3906 closed 2 years ago

YangWenhao3906 commented 3 years ago

I'm tring to run the code in an essay but get Error about install pytorch-mpi when doing environment configuration.

My Ubuntu is 16.04.

I ran the Docker file the essay provided to do environment configuration.

This is the source code https://github.com/epfml/federated-learning-public-code

I tried two times but got the same result. 

Service 'pytorch-mpi' failed to build: The command '/bin/sh -c cd pytorch &&     git checkout v1.6.0 &&     git submodule sync &&     git submodule update --init &&     TORCH_CUDA_ARCH_LIST="3.5 3.7 5.2 6.0 6.1 7.0+PTX" TORCH_NVCC_FLAGS="-Xfatbin -compress-all"     CMAKE_PREFIX_PATH="$(dirname $(which $HOME/conda/bin/conda))/../"     pip install -v .' returned a non-zero code: 1

I paste pytorch-mpi Dockerfile below.

FROM user/base-db

USER $NB_USER
WORKDIR $HOME

# install openMPI
RUN mkdir $HOME/.openmpi/
RUN wget https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.gz
RUN gunzip -c openmpi-3.0.0.tar.gz | tar xf - \
    && cd openmpi-3.0.0 \
    && ./configure --prefix=$HOME/.openmpi/ --with-cuda \
    && make all install

ENV PATH $HOME/.openmpi/bin:$PATH
ENV LD_LIBRARY_PATH $HOME/.openmpi/lib:$LD_LIBRARY_PATH

# install conda
ENV PYTHON_VERSION=3.6
RUN curl -o ~/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
    sh ~/miniconda.sh -b -p $HOME/conda && \
    rm ~/miniconda.sh
RUN $HOME/conda/bin/conda update -n base conda
RUN $HOME/conda/bin/conda create -y --name pytorch-py$PYTHON_VERSION python=$PYTHON_VERSION numpy pyyaml scipy ipython mkl mkl-include
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION -c soumith magma-cuda100
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION scikit-learn
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install pytelegraf pymongo influxdb kubernetes jinja2
ENV PATH $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin:$PATH

# install pytorch, torchvision, torchtext.
RUN git clone --recursive  https://github.com/pytorch/pytorch
RUN cd pytorch && \
    git checkout v1.6.0 && \
    git submodule sync && \
    git submodule update --init && \
    TORCH_CUDA_ARCH_LIST="3.5 3.7 5.2 6.0 6.1 7.0+PTX" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
    CMAKE_PREFIX_PATH="$(dirname $(which $HOME/conda/bin/conda))/../" \
    pip install -v .
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION  av -c conda-forge
RUN git clone https://github.com/pytorch/vision.git && cd vision && git checkout v0.7.0 && python setup.py install
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install --upgrade git+https://github.com/pytorch/text
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install nltk && $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/python -c "import nltk; nltk.download('wordnet')"
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install torchcontrib gpytorch
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install spacy
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/python -m spacy download en
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/python -m spacy download de

# install bit2byte.
RUN git clone https://github.com/tvogels/signSGD-with-Majority-Vote.git && \
    cd signSGD-with-Majority-Vote/main/bit2byte-extension/ && \
    $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/python setup.py develop --user

# install other python related softwares.
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION -y opencv protobuf
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION -y networkx
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION -y -c anaconda pandas
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION -y -c conda-forge tabulate
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install lmdb tensorboard_logger pyarrow msgpack msgpack_numpy mpi4py
RUN $HOME/conda/envs/pytorch-py$PYTHON_VERSION/bin/pip install POT opencv-python
RUN $HOME/conda/bin/conda install --name pytorch-py$PYTHON_VERSION -c conda-forge python-blosc
RUN $HOME/conda/bin/conda clean -ya

I paste the shell output below.

[ 48%] Built target dnnl_cpu_x64
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_u8.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_i32.o
    [ 48%] Built target fbgemm_avx2
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_u32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_i64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_u64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_f16.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_f32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_prod_f64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_i8.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_u8.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_i32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_u32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_i64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_u64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_f16.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_f32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_min_f64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_i8.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_u8.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_i32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_u32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_i64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_u64.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_f16.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_f32.o
    Compiling  all_reduce.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_reduce_max_f64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_i8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_u8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_i32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_u32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_i64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_u64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_f16.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_f32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_sum_f64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_i8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_u8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_i32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_u32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_i64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_u64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_f16.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_f32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_prod_f64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_i8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_u8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_i32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_u32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_i64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_u64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_f16.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_f32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_min_f64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_i8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_u8.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_i32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_u32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_i64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_u64.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_f16.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_f32.o
    Compiling  all_gather.cu                       > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/all_gather_max_f64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_i8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_u8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_i32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_u32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_i64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_u64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_f16.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_f32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_sum_f64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_i8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_u8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_i32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_u32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_i64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_u64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_f16.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_f32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_prod_f64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_i8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_u8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_i32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_u32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_i64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_u64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_f16.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_f32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_min_f64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_i8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_u8.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_i32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_u32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_i64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_u64.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_f16.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_f32.o
    Compiling  broadcast.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/broadcast_max_f64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_i8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_u8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_i32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_u32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_i64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_u64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_f16.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_f32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_sum_f64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_i8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_u8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_i32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_u32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_i64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_u64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_f16.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_f32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_prod_f64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_i8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_u8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_i32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_u32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_i64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_u64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_f16.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_f32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_min_f64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_i8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_u8.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_i32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_u32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_i64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_u64.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_f16.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_f32.o
    Compiling  reduce.cu                           > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_max_f64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_i8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_u8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_i32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_u32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_i64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_u64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_f16.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_f32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_sum_f64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_i8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_u8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_i32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_u32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_i64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_u64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_f16.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_f32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_prod_f64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_i8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_u8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_i32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_u32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_i64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_u64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_f16.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_f32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_min_f64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_i8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_u8.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_i32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_u32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_i64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_u64.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_f16.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_f32.o
    Compiling  reduce_scatter.cu                   > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/reduce_scatter_max_f64.o
    Compiling  functions.cu                        > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/functions.o
    Archiving  objects                             > /tmp/pip-req-build-vig33zxc/build/nccl/obj/collectives/device/colldevice.a
    Linking    libnccl.so.2.4.8                    > /tmp/pip-req-build-vig33zxc/build/nccl/lib/libnccl.so.2.4.8
    Generating nccl.pc.in                          > /tmp/pip-req-build-vig33zxc/build/nccl/lib/pkgconfig/nccl.pc
    Archiving  libnccl_static.a                    > /tmp/pip-req-build-vig33zxc/build/nccl/lib/libnccl_static.a
    /tmp/pip-req-build-vig33zxc/third_party/nccl/nccl/src
    [ 48%] No install step for 'nccl_external'
    [ 48%] Completed 'nccl_external'
    [ 48%] Built target nccl_external
    Makefile:138: recipe for target 'all' failed
    make: *** [all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-vig33zxc/setup.py", line 732, in <module>
        build_deps()
      File "/tmp/pip-req-build-vig33zxc/setup.py", line 316, in build_deps
        cmake=cmake)
      File "/tmp/pip-req-build-vig33zxc/tools/build_pytorch_libs.py", line 62, in build_caffe2
        cmake.build(my_env)
      File "/tmp/pip-req-build-vig33zxc/tools/setup_helpers/cmake.py", line 345, in build
        self.run(build_args, my_env)
      File "/tmp/pip-req-build-vig33zxc/tools/setup_helpers/cmake.py", line 141, in run
        check_call(command, cwd=self.build_dir, env=env)
      File "/home/user/conda/envs/pytorch-py3.6/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 2.
    Running setup.py install for torch: finished with status 'error'
ERROR: Command errored out with exit status 1: /home/user/conda/envs/pytorch-py3.6/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-vig33zxc/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-vig33zxc/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-jts14nkl/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/conda/envs/pytorch-py3.6/include/python3.6m/torch Check the logs for full command output.
ERROR: Service 'pytorch-mpi' failed to build: The command '/bin/sh -c cd pytorch &&     git checkout v1.6.0 &&     git submodule sync &&     git submodule update --init &&     TORCH_CUDA_ARCH_LIST="3.5 3.7 5.2 6.0 6.1 7.0+PTX" TORCH_NVCC_FLAGS="-Xfatbin -compress-all"     CMAKE_PREFIX_PATH="$(dirname $(which $HOME/conda/bin/conda))/../"     pip install -v .' returned a non-zero code: 1
its-Ahmad commented 7 months ago

I am having the exact same problem... Any solutions?