Trouble while cloning the repository.

anthonymakela commented 5 years ago

Hi! There seems to be a problem while cloning the repository. Im trying to build with multiple GPU support and this seems to be the root of the problem. Any ideas on how to fix this?

fatal: reference is not a tree: e1c8056f6a0ee1c42fd00430b74176e67db66a9f Unable to checkout 'e1c8056f6a0ee1c42fd00430b74176e67db66a9f' in submodule path 'rabit'

hcho3 commented 5 years ago

What is your Git version? I cannot reproduce the error:

ubuntu@ip-172-31-1-97:~$ git --version
git version 2.17.1
ubuntu@ip-172-31-1-97:~$ git clone --recursive https://github.com/dmlc/xgboost
Cloning into 'xgboost'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 29649 (delta 3), reused 8 (delta 2), pack-reused 29634
Receiving objects: 100% (29649/29649), 12.15 MiB | 37.80 MiB/s, done.
Resolving deltas: 100% (17171/17171), done.
Submodule 'cub' (https://github.com/NVlabs/cub) registered for path 'cub'
Submodule 'dmlc-core' (https://github.com/dmlc/dmlc-core) registered for path 'dmlc-core'
Submodule 'rabit' (https://github.com/dmlc/rabit) registered for path 'rabit'
Cloning into '/home/ubuntu/xgboost/cub'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 32675 (delta 0), reused 4 (delta 0), pack-reused 32671
Receiving objects: 100% (32675/32675), 16.52 MiB | 45.71 MiB/s, done.
Resolving deltas: 100% (28643/28643), done.
Cloning into '/home/ubuntu/xgboost/dmlc-core'...
remote: Enumerating objects: 29, done.
remote: Counting objects: 100% (29/29), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 5677 (delta 6), reused 20 (delta 5), pack-reused 5648
Receiving objects: 100% (5677/5677), 1.43 MiB | 27.67 MiB/s, done.
Resolving deltas: 100% (3429/3429), done.
Cloning into '/home/ubuntu/xgboost/rabit'...
remote: Enumerating objects: 3345, done.
remote: Total 3345 (delta 0), reused 0 (delta 0), pack-reused 3345
Receiving objects: 100% (3345/3345), 980.42 KiB | 21.31 MiB/s, done.
Resolving deltas: 100% (2178/2178), done.
Submodule path 'cub': checked out 'b20808b1b04ec3d6a625e51fbc1eb76f337754ad'
Submodule path 'dmlc-core': checked out '7245c9d42e0baf3b030ae580c125f403dbe57c9f'
remote: Enumerating objects: 1, done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 1
Unpacking objects: 100% (1/1), done.
From https://github.com/dmlc/rabit
 * branch            e1c8056f6a0ee1c42fd00430b74176e67db66a9f -> FETCH_HEAD
Submodule path 'rabit': checked out 'e1c8056f6a0ee1c42fd00430b74176e67db66a9f'

trivialfis commented 5 years ago

With git 2.22.0 I cannot reproduce.

paantya commented 5 years ago

Hi!

Аналогично:

Cloning into 'rabit'...
fatal: reference is not a tree: e1c8056f6a0ee1c42fd00430b74176e67db66a9f
Unable to checkout 'e1c8056f6a0ee1c42fd00430b74176e67db66a9f' in submodule path 'rabit'

i use Docker:

##########################################################
# Dockerfile to build Data Science Python Toolbox
# Based on latest LTS release ubuntu:xenial
##########################################################

## Set the base image to debian
FROM ubuntu:16.04

## File author
LABEL maintainer="user"

## Version
ENV REFRESHED_AT 2019-01-06

## Specify user
USER root

## No tty
ENV DEBIAN_FRONTEND noninteractive

## Configure locale and environment variables
RUN apt-get update && apt-get install -y apt-utils locales && locale-gen en_US.UTF-8 && dpkg-reconfigure locales

ENV HOME /root
ENV WORKDIR /root/shared

## Setup work directory
RUN mkdir $WORKDIR

## Set current dir
WORKDIR $HOME

## Install compilers and essential tools
RUN apt-get update && apt-get install --no-install-recommends -y \
    gcc \
    g++ \
    gfortran \
    build-essential \
    software-properties-common \
    pkg-config \
    cmake \
    vim \
    nano \
    tar \
    unzip \
    git \
    curl \
    wget \
    python3 \
    python3-dev \
    python3-pip \
    net-tools \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/*

## Install dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
    python3-tk \
    libpng-dev \
    libfreetype6-dev \
    libxft-dev \
    libjpeg-dev \
    libhdf5-dev \
    libffi-dev \
    libssl-dev \
    libatlas-dev \
    libatlas-base-dev \
    liblapack-dev \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/*

## Install scipy stack
RUN pip3 install --upgrade --ignore-installed pip setuptools wheel
RUN pip3 install \
    numpy \
    scipy \
    sympy \
    matplotlib \
    seaborn \
    pandas \
    hyperopt \
    shap \
    dask \
    slacker \
    tqdm \
    ipyparallel \
    jupyter_contrib_nbextensions \
    telepyth \ 
    statsmodels \
    nose \
    cython \
    h5py \
    pillow \
    scikit-learn \
    glmnet_py \
    cvxopt \
    pystan \
    ipykernel \
    ipython \
    jupyter \
    notebook \
    jupyterlab \
    bokeh \
    holoviews \
    ipywidgets \
    ipyvolume \
    joblib \
    arrow \
    numba \
    pytz \
    graphviz \
    lightgbm \
    tensorflow \
    pyarrow \
    feather-format \
    ptvsd \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* $HOME/.cache/pip/*

##install --upgrade tables
RUN pip3 install --upgrade --ignore-installed pip setuptools wheel
RUN pip3 install \
    dask[dataframe] --upgrade \
    --upgrade tables \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* $HOME/.cache/pip/*

## Install XGBoost library
RUN git clone --recursive https://github.com/dmlc/xgboost && cd xgboost && make -j4 && cd python-package && python3 setup.py install

terminal:

Step 18/30 : RUN git clone --recursive https://github.com/dmlc/xgboost && cd xgboost && make -j4 && cd python-package && python3 setup.py install
 ---> Running in 0d17bf33e241
Cloning into 'xgboost'...
Submodule 'cub' (https://github.com/NVlabs/cub) registered for path 'cub'
Submodule 'dmlc-core' (https://github.com/dmlc/dmlc-core) registered for path 'dmlc-core'
Submodule 'rabit' (https://github.com/dmlc/rabit) registered for path 'rabit'
Cloning into 'cub'...
Submodule path 'cub': checked out 'b20808b1b04ec3d6a625e51fbc1eb76f337754ad'
Cloning into 'dmlc-core'...
Submodule path 'dmlc-core': checked out 'b46747af11336e8a138322139a45ee1dfe64e754'
Cloning into 'rabit'...
fatal: reference is not a tree: e1c8056f6a0ee1c42fd00430b74176e67db66a9f
Unable to checkout 'e1c8056f6a0ee1c42fd00430b74176e67db66a9f' in submodule path 'rabit'
The command '/bin/sh -c git clone --recursive https://github.com/dmlc/xgboost && cd xgboost && make -j4 && cd python-package && python3 setup.py install' returned a non-zero code: 1

trivialfis commented 5 years ago

Could you specify git version?

paantya commented 5 years ago

Could you specify git version?

Step 12/18 : RUN git --version
 ---> Running in 34fece459459
git version 2.7.4

paantya commented 5 years ago

@trivialfis you can run my script in terminal: user@ubuntu:~/mlbox$ ./build.sh

Where build.sh: docker build -t user:cpu -f Dockerfile.cpu .

And Dockerfile.cpu:

##########################################################
# Dockerfile to build Data Science Python Toolbox
# Based on latest LTS release ubuntu:xenial
##########################################################

## Set the base image to debian
FROM ubuntu:16.04

## File author
LABEL maintainer="user"

## Version
ENV REFRESHED_AT 2019-07-26

## Specify user
USER root

## No tty
ENV DEBIAN_FRONTEND noninteractive

## Configure locale and environment variables
RUN apt-get update && apt-get install -y apt-utils locales && locale-gen en_US.UTF-8 && dpkg-reconfigure locales

ENV HOME /root
ENV WORKDIR /root/shared

## Setup work directory
RUN mkdir $WORKDIR

## Set current dir
WORKDIR $HOME

## Install compilers and essential tools
RUN apt-get update && apt-get install --no-install-recommends -y \
    gcc \
    g++ \
    gfortran \
    build-essential \
    software-properties-common \
    pkg-config \
    cmake \
    vim \
    nano \
    tar \
    unzip \
    git \
    curl \
    wget \
    python3 \
    python3-dev \
    python3-pip \
    net-tools \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/*

## check git vers
RUN git --version

## Install dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
    python3-tk \
    libpng-dev \
    libfreetype6-dev \
    libxft-dev \
    libjpeg-dev \
    libhdf5-dev \
    libffi-dev \
    libssl-dev \
    libatlas-dev \
    libatlas-base-dev \
    liblapack-dev \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/*

## Install scipy stack
RUN pip3 install --upgrade --ignore-installed pip setuptools wheel
RUN pip3 install \
    numpy \
    scipy \
    sympy \
    matplotlib \
    seaborn \
    pandas \
    hyperopt \
    shap \
    dask \
    slacker \
    tqdm \
    ipyparallel \
    jupyter_contrib_nbextensions \
    telepyth \ 
    statsmodels \
    nose \
    cython \
    h5py \
    pillow \
    scikit-learn \
    glmnet_py \
    cvxopt \
    pystan \
    ipykernel \
    ipython \
    jupyter \
    notebook \
    jupyterlab \
    bokeh \
    holoviews \
    ipywidgets \
    ipyvolume \
    joblib \
    arrow \
    numba \
    pytz \
    graphviz \
    lightgbm \
    tensorflow \
    pyarrow \
    feather-format \
    ptvsd \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* $HOME/.cache/pip/*

##install --upgrade tables
RUN pip3 install --upgrade --ignore-installed pip setuptools wheel
RUN pip3 install \
    dask[dataframe] --upgrade \
    --upgrade tables \
    && apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* $HOME/.cache/pip/*

## Install XGBoost library
RUN git clone --recursive https://github.com/dmlc/xgboost && cd xgboost && make -j4 && cd python-package && python3 setup.py install

anthonymakela commented 5 years ago

This error was due to Azure's Linux machines. By default they seem to have an really old git version. Updating git solved the problem.

paantya commented 5 years ago

@anthonymakela but i don't use Azure's Linux.

something else?

anthonymakela commented 5 years ago

Can you try updating your git version to >= 2.20? Im pretty confident that it will fix the issue. @paantya

jpbowman01 commented 5 years ago

Joining in this thread as I have the same problem... unfortunately, I don't have the ability to update git - 2.7.4 is the latest version on Ubuntu 16.04, which is what I'm running, and I'm not the sysadmin (a work machine.) Any other thoughts?

trivialfis commented 5 years ago

For now you can just use a release source from github release page. I'm not sure about the cause of this.

trivialfis commented 5 years ago

Could you please help testing the latest master branch? @jpbowman01 @anthonymakela @paantya

paantya commented 5 years ago

@anthonymakela updating git and it work!

Add in Docker before install git:

##Get last git
RUN apt-get update

RUN apt-get install software-properties-common -y 

RUN add-apt-repository ppa:git-core/ppa -y

@jpbowman01 you can try:

sudo apt-get update
sudo apt-get install software-properties-common -y 
sudo add-apt-repository ppa:git-core/ppa -y
sudo apt-get install git

upd: I tried it 27 hours ago

paantya commented 5 years ago

Could you please help testing the latest master branch? @jpbowman01 @anthonymakela @paantya

You about at git < 2.20 testing the latest master branch?

trivialfis commented 5 years ago

@paantya Right. See if the problem persists with older version of git (the one you used before).

paantya commented 5 years ago

@trivialfis all ok! i run Docker file (look up top), and it finished ok: > Finished processing dependencies for xgboost===1.0.0-SNAPSHOT git 2.7.4

trivialfis commented 5 years ago

@paantya Great news! Thanks.

dmlc / xgboost

Trouble while cloning the repository. #4685