search

microsoft / LightGBM

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
https://lightgbm.readthedocs.io/en/latest/
MIT License
16.67k stars 3.83k forks source link

Broken link - Failed to clone 'external_libs/eigen' #4230

Closed jfrabetti closed 3 years ago

jfrabetti commented 3 years ago

Description

I am trying to create a wheel by installing with the --nopm option for AWS Glue (OpenMP not supported) using these directions: https://pypi.org/project/lightgbm/

Install from GitHub
git clone --recursive https://github.com/microsoft/LightGBM.git
cd LightGBM/python-package
 # export CXX=g++-7 CC=gcc-7  # macOS users, if you decided to compile with gcc, don't forget to specify compilers (replace "7" with version of gcc installed on your machine)
python setup.py install

Reproducible example

Fails on this line:

git clone --recursive https://github.com/microsoft/LightGBM.git

Environment info

wsl Ubuntu 20.04 also failed:

  pip install lightgbm --install-option=--nomp

Additional Comments

Here is a screenshot that shows the external_lib is not linked to anything / broken link:

broken
jameslamb commented 3 years ago

Hi @jfrabetti , thanks for using LightGBM!

I don't believe that the fact that eigen isn't a clickable link is problematic. I think it is always like that because eigen is hosted on GitLab, and GitHub only renders those links for submodules that point to other GitHub repos.

You can see the sources of the submodules at https://github.com/microsoft/LightGBM/blob/5014f19f150e5a6ebe236584c52519ee11446f8a/.gitmodules.

I double-checked https://status.gitlab.com/ and there haven't been any recent outages. I also tried cloning the repository myself a moment ago and didn't have any issues.

git clone --recursive https://github.com/microsoft/LightGBM.git
cd python-package
python setup.py install

I was also able to install from PyPI without issue (on Ubuntu 18.04).

pip install lightgbm --install-option=--nomp

So I think we need more information. Could you please tell me what specifically you mean when you say that cloning "fails"? For example, could you please share any logs that were printed?

jfrabetti commented 3 years ago

I see...then it probably has something to do my proxy settings, but here is the error:

Cloning into 'LightGBM'...
remote: Enumerating objects: 21864, done.
remote: Counting objects: 100% (607/607), done.
remote: Compressing objects: 100% (358/358), done.
remote: Total 21864 (delta 367), reused 410 (delta 238), pack-reused 21257
Receiving objects: 100% (21864/21864), 17.22 MiB | 7.05 MiB/s, done.
Resolving deltas: 100% (15936/15936), done.
Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'external_libs/compute'
Submodule 'eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'external_libs/eigen'
Submodule 'external_libs/fast_double_parser' (https://github.com/lemire/fast_double_parser.git) registered for path 'external_libs/fast_double_parser'
Submodule 'external_libs/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'external_libs/fmt'
Cloning into '/home/frabetti/projects/lightgbm/LightGBM/external_libs/compute'...
remote: Enumerating objects: 21728, done.
remote: Total 21728 (delta 0), reused 0 (delta 0), pack-reused 21728
Receiving objects: 100% (21728/21728), 8.51 MiB | 4.79 MiB/s, done.
Resolving deltas: 100% (17566/17566), done.
Cloning into '/home/frabetti/projects/lightgbm/LightGBM/external_libs/eigen'...
remote: Enumerating objects: 890, done.
remote: Counting objects: 100% (890/890), done.
remote: Compressing objects: 100% (312/312), done.
remote: Total 109524 (delta 687), reused 753 (delta 575), pack-reused 108634
Receiving objects: 100% (109524/109524), 101.72 MiB | 4.49 MiB/s, done.
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
Resolving deltas: 100% (89954/89954), done.
fatal: clone of 'https://gitlab.com/libeigen/eigen.git' into submodule path '/home/frabetti/projects/lightgbm/LightGBM/external_libs/eigen' failed
Failed to clone 'external_libs/eigen'. Retry scheduled
Cloning into '/home/frabetti/projects/lightgbm/LightGBM/external_libs/fast_double_parser'...
remote: Enumerating objects: 675, done.
remote: Counting objects: 100% (175/175), done.
remote: Compressing objects: 100% (110/110), done.
remote: Total 675 (delta 86), reused 94 (delta 39), pack-reused 500
Receiving objects: 100% (675/675), 799.30 KiB | 2.28 MiB/s, done.
Resolving deltas: 100% (340/340), done.
Cloning into '/home/frabetti/projects/lightgbm/LightGBM/external_libs/fmt'...
remote: Enumerating objects: 25192, done.
remote: Counting objects: 100% (364/364), done.
remote: Compressing objects: 100% (172/172), done.
remote: Total 25192 (delta 199), reused 260 (delta 149), pack-reused 24828
Receiving objects: 100% (25192/25192), 12.82 MiB | 6.77 MiB/s, done.
Resolving deltas: 100% (17010/17010), done.
Cloning into '/home/frabetti/projects/lightgbm/LightGBM/external_libs/eigen'...
remote: Enumerating objects: 890, done.
remote: Counting objects: 100% (890/890), done.
remote: Compressing objects: 100% (312/312), done.
remote: Total 109524 (delta 687), reused 753 (delta 575), pack-reused 108634
Receiving objects: 100% (109524/109524), 101.72 MiB | 2.23 MiB/s, done.
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
Resolving deltas: 100% (89954/89954), done.
fatal: clone of 'https://gitlab.com/libeigen/eigen.git' into submodule path '/home/frabetti/projects/lightgbm/LightGBM/external_libs/eigen' failed
Failed to clone 'external_libs/eigen' a second time, aborting
jameslamb commented 3 years ago

Thanks for that!

Could you also please share any logs you get from running the following?

pip install lightgbm --install-option=--nomp

The source distributions available on PyPI already have all of the relevant code from these submodules bundled in them, so there shouldn't be any dependency on cloning repos with that option.

For the case where you're cloning LightGBM, these lines in the log look problematic:

error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.

It's interesting that only Eigen failed, and not the others. It is different in at least two ways from the other submodules:

As a test, you could try shelling into whatever environment this is running in and directly cloning something else from GitLab, like

git clone https://gitlab.com/vigou3/actuar.git

If that fails, you probably have a networking problem preventing cloning from GitLab into your environment.

If that works, then you'll maybe there is something else specific to eigen (such as its size) that is affecting cloning.

If eigen's size is a problem, you could try cloning the submodules manually and limiting how much of their history you clone. Like this:

git clone --depth 1 https://github.com/microsoft/LightGBM.git
cd LightGBM

rm .gitmodules

git clone --depth 1 https://github.com/boostorg/compute.git external_libs/compute/
git clone --depth 1 https://gitlab.com/libeigen/eigen.git external_libs/eigen
git clone --depth 1 https://github.com/fmtlib/fmt.git external_libs/fmt/
git clone --depth 1 https://github.com/lemire/fast_double_parser.git external_libs/fast_double_parser

cd python-package
python setup.py install

By the way, I edited your comment slightly to format it as a code block, as I think that's a bit easier to read and visually separate from the regular text. In case you're not familiar with doing that on GitHub, I use three backticks at the beginning and end of the block, like this

image

You can find more details on that at https://docs.github.com/en/github/writing-on-github/basic-writing-and-formatting-syntax.

jameslamb commented 3 years ago

I hope that'll help. If you do end up solving this, please do come back and leave a comment here explaining how you resolved that problem. That will be helpful for anyone else experiencing a similar issue who finds this from search.

jfrabetti commented 3 years ago

Thank you for formatting my code. I am able to clone this repo / execute this command 'git clone https://gitlab.com/vigou3/actuar.git'. There is a huge error output, but here is the end for the pip install:

 File "/home/frabetti/virtualenv/nomp/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 500, in _prepare_linked_requirement
      dist = _get_prepared_distribution(
    File "/home/frabetti/virtualenv/nomp/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 65, in _get_prepared_distribution
      with req_tracker.track(req):
    File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
      return next(self.gen)
    File "/home/frabetti/virtualenv/nomp/lib/python3.8/site-packages/pip/_internal/req/req_tracker.py", line 144, in track
      self.add(req)
    File "/home/frabetti/virtualenv/nomp/lib/python3.8/site-packages/pip/_internal/req/req_tracker.py", line 110, in add
      raise LookupError(message)
  LookupError: https://files.pythonhosted.org/packages/fe/fd/8704c7b7b34cdac850485e638346025ca57c5a859934b9aa1be5399b33b7/scipy-1.6.3.tar.gz#sha256=a75b014d3294fce26852a9d04ea27b5671d86736beb34acdfc05859246260707 (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10) is already being built: scipy from https://files.pythonhosted.org/packages/fe/fd/8704c7b7b34cdac850485e638346025ca57c5a859934b9aa1be5399b33b7/scipy-1.6.3.tar.gz#sha256=a75b014d3294fce26852a9d04ea27b5671d86736beb34acdfc05859246260707 (from lightgbm)
  WARNING: You are using pip version 21.0.1; however, version 21.1 is available.
  You should consider upgrading via the '/home/frabetti/virtualenv/nomp/bin/python -m pip install --upgrade pip' command.
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/72/e4/75247cf75e9e3ba1bf296c9d26ba1cfc71d781821d32bcdb4d9b4b1b4153/scikit-learn-0.23.0.tar.gz#sha256=639a53df6273acc6a7510fb0c658b94e0c70bb13dafff9d14932c981ff9baff4 (from https://pypi.org/simple/scikit-learn/) (requires-python:>=3.6). Command errored out with exit status 2: /home/frabetti/virtualenv/nomp/bin/python /home/frabetti/virtualenv/nomp/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rrsauqqd/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1' Check the logs for full command output.
  Using cached scikit-learn-0.22.2.post1.tar.gz (6.9 MB)
Collecting joblib>=0.11
  Using cached joblib-1.0.1.tar.gz (316 kB)
Skipping wheel build for lightgbm, due to binaries being disabled for it.
Skipping wheel build for scikit-learn, due to binaries being disabled for it.
Skipping wheel build for joblib, due to binaries being disabled for it.
Skipping wheel build for scipy, due to binaries being disabled for it.
Installing collected packages: scipy, joblib, scikit-learn, lightgbm
    Running setup.py install for scipy ... error
    ERROR: Command errored out with exit status 1:
     command: /home/frabetti/virtualenv/nomp/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o99mdhpq/scipy_469cd2c1c0724fb084e059981c4eee2f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-o99mdhpq/scipy_469cd2c1c0724fb084e059981c4eee2f/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8lvmgi8x/install-record.txt --single-version-externally-managed --compile --install-headers /home/frabetti/virtualenv/nomp/include/site/python3.8/scipy --nomp
         cwd: /tmp/pip-install-o99mdhpq/scipy_469cd2c1c0724fb084e059981c4eee2f/
    Complete output (15 lines):

    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:

      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install scipy`   (last SciPy release on PyPI)

    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --nomp not recognized
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/frabetti/virtualenv/nomp/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-o99mdhpq/scipy_469cd2c1c0724fb084e059981c4eee2f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-o99mdhpq/scipy_469cd2c1c0724fb084e059981c4eee2f/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8lvmgi8x/install-record.txt --single-version-externally-managed --compile --install-headers /home/frabetti/virtualenv/nomp/include/site/python3.8/scipy --nomp Check the logs for full command output.

I used my personal laptop so I am all set if you want to close this.

jameslamb commented 3 years ago

Ah, interesting! That second error looks like it is coming from the fact that you ended up getting a source installation of scipy instead of a wheel when you ran

pip install lightgbm --install-option=--nomp

Skipping wheel build for scipy, due to binaries being disabled for it.

I can see the source for that error message at https://github.com/scipy/scipy/blob/f97ce31736bb3c4a0a0c7db8fde2ee4ebc69e39d/setup.py#L395-L405

I expect that if you do pip install scipy in that environment, you'll get the same error. Maybe you are on a combination of operating system / Python version that doesn't match any of the wheels at https://pypi.org/project/scipy/#files. So in that case, it looks like installing lightgbm failed because of an error installing scipy.

StrikerRUS commented 3 years ago

From my personal experience libcurl4-openssl works much better out-of-the-box than libcurl4-gnutls-dev.

StrikerRUS commented 3 years ago

Ah, the first result from googling curl 56 GnuTLS recv error (-110) agreed with me 😄 https://stackoverflow.com/a/40465312

jameslamb commented 3 years ago

Oh wow, good find @StrikerRUS !

jameslamb commented 3 years ago

It seems like this can be closed.

@jfrabetti come back any time if you have other questions!

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.