search

DmitryUlyanov / Multicore-TSNE

Parallel t-SNE implementation with Python and Torch wrappers.
Other
1.88k stars 229 forks source link

Error in builds using cmake >= 3.20? #81

Open jreades opened 3 years ago

jreades commented 3 years ago

I'm trying to install MulticoreTSNE into a Docker image built on top of the Jupyter Minimal distribution (Anaconda Python, etc.). Previously I was able to run this without a hitch. I've tested the following combinations:

  1. jupyter/minimal-notebook:584f43f0658 (Late April 2021)
    • cmake 3.18.2 = Build successful (this was the version dumped from my last working image)
    • cmake 3.20.1 = Build unsuccessful (error below)
  2. jupyter/minimal-notebook:016833b15ceb (Late Feb 2021)
    • cmake 3.18.2 = Build successful (this was the version dumped from my last working image)
    • cmake 3.20.1 = Build unsuccessful (error below)

Forcing cmake to downgrade to 3.18.2 seems to force a downgrade of Python from 3.9 to 3.8 so it's possible that is the source of the problem, but the report suggests it's cmake.

The error is:

#13 53.91     running build_ext
#13 53.91     cmake version 3.20.1
#13 53.91
#13 53.91     CMake suite maintained and supported by Kitware (kitware.com/cmake).
#13 53.91     CMake Error: Unknown argument --
#13 53.91     CMake Error: Run 'cmake --help' for all supported options.
#13 53.91
#13 53.91     ERROR: Cannot generate Makefile. See above errors.

Full context is:

> [4/5] RUN conda-env create -n ethos -f ./python.test.yml     && conda clean --all --yes --force-pkgs-dirs     && find /opt/conda/ -follow -type f -name '*.a' -delete     && find /opt/conda/ -follow -type f -name '*.pyc' -delete     && find /opt/conda/ -follow -type f -name '*.js.map' -delete     && pip cache purge     && rm -rf /home/jovyan/.cache/pip     && rm ./python.test.yml:
#12 0.549 Collecting package metadata (repodata.json): ...working... done
#12 25.27 Solving environment: ...working... done
#12 29.94
#12 29.94 Downloading and Extracting Packages
libffi-3.3           | 51 KB     | ########## | 100%
lz4-c-1.9.3          | 179 KB    | ########## | 100%
libgomp-9.3.0        | 376 KB    | ########## | 100%
libedit-3.1.20191231 | 121 KB    | ########## | 100%
xz-5.2.5             | 343 KB    | ########## | 100%
rhash-1.4.1          | 192 KB    | ########## | 100%
krb5-1.17.2          | 1.4 MB    | ########## | 100%
readline-8.1         | 295 KB    | ########## | 100%
bzip2-1.0.8          | 484 KB    | ########## | 100%
_openmp_mutex-4.5    | 22 KB     | ########## | 100%
tzdata-2021a         | 121 KB    | ########## | 100%
libssh2-1.9.0        | 226 KB    | ########## | 100%
certifi-2020.12.5    | 143 KB    | ########## | 100%
sqlite-3.35.4        | 1.4 MB    | ########## | 100%
ca-certificates-2020 | 137 KB    | ########## | 100%
zstd-1.4.9           | 431 KB    | ########## | 100%
libuv-1.41.0         | 1.0 MB    | ########## | 100%
setuptools-49.6.0    | 943 KB    | ########## | 100%
libstdcxx-ng-9.3.0   | 4.0 MB    | ########## | 100%
libcurl-7.76.1       | 328 KB    | ########## | 100%
cmake-3.20.1         | 14.7 MB   | ########## | 100%
libgcc-ng-9.3.0      | 7.8 MB    | ########## | 100%
python-3.9.2         | 27.3 MB   | ########## | 100%
zlib-1.2.11          | 106 KB    | ########## | 100%
pip-21.0.1           | 1.1 MB    | ########## | 100%
ld_impl_linux-64-2.3 | 618 KB    | ########## | 100%
tk-8.6.10            | 3.2 MB    | ########## | 100%
openssl-1.1.1k       | 2.1 MB    | ########## | 100%
libnghttp2-1.43.0    | 808 KB    | ########## | 100%
wheel-0.36.2         | 31 KB     | ########## | 100%
python_abi-3.9       | 4 KB      | ########## | 100%
libev-4.33           | 104 KB    | ########## | 100%
_libgcc_mutex-0.1    | 3 KB      | ########## | 100%
expat-2.3.0          | 168 KB    | ########## | 100%
ncurses-6.2          | 985 KB    | ########## | 100%
c-ares-1.17.1        | 109 KB    | ########## | 100%
#12 44.17 Preparing transaction: ...working... done
#12 44.43 Verifying transaction: ...working... done
#12 45.95 Executing transaction: ...working... done
#12 47.59 Installing pip dependencies: ...working... Ran pip subprocess with arguments:
#12 52.31 ['/opt/conda/envs/ethos/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/jovyan/condaenv.4kzuzpkn.requirements.txt']
#12 52.31 Pip subprocess output:
#12 52.31 Collecting MulticoreTSNE
#12 52.31   Downloading MulticoreTSNE-0.1.tar.gz (20 kB)
#12 52.31 Collecting numpy
#12 52.31   Downloading numpy-1.20.2-cp39-cp39-manylinux2010_x86_64.whl (15.4 MB)
#12 52.31 Collecting cffi
#12 52.31   Downloading cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl (406 kB)
#12 52.31 Collecting pycparser
#12 52.31   Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
#12 52.31 Building wheels for collected packages: MulticoreTSNE
#12 52.31   Building wheel for MulticoreTSNE (setup.py): started
#12 52.31   Building wheel for MulticoreTSNE (setup.py): finished with status 'error'
#12 52.31   Running setup.py clean for MulticoreTSNE
#12 52.31 Failed to build MulticoreTSNE
#12 52.31 Installing collected packages: pycparser, numpy, cffi, MulticoreTSNE
#12 52.31     Running setup.py install for MulticoreTSNE: started
#12 52.31     Running setup.py install for MulticoreTSNE: finished with status 'error'
#12 52.31
#12 52.31 failed
#12 52.31
#12 52.31
#12 52.31 ==> WARNING: A newer version of conda exists. <==
#12 52.31   current version: 4.10.0
#12 52.31   latest version: 4.10.1
#12 52.31
#12 52.31 Please update conda by running
#12 52.31
#12 52.31     $ conda update -n base conda
#12 52.31
#12 52.31
#12 52.31 Pip subprocess error:
#12 52.31   ERROR: Command errored out with exit status 1:
#12 52.31    command: /opt/conda/envs/ethos/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gm7zmxgs
#12 52.31        cwd: /tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/
#12 52.31   Complete output (26 lines):
#12 52.31   running bdist_wheel
#12 52.31   running build
#12 52.31   running build_py
#12 52.31   creating build
#12 52.31   creating build/lib.linux-x86_64-3.9
#12 52.31   creating build/lib.linux-x86_64-3.9/MulticoreTSNE
#12 52.31   copying MulticoreTSNE/__init__.py -> build/lib.linux-x86_64-3.9/MulticoreTSNE
#12 52.31   creating build/lib.linux-x86_64-3.9/MulticoreTSNE/tests
#12 52.31   copying MulticoreTSNE/tests/test_base.py -> build/lib.linux-x86_64-3.9/MulticoreTSNE/tests
#12 52.31   copying MulticoreTSNE/tests/__init__.py -> build/lib.linux-x86_64-3.9/MulticoreTSNE/tests
#12 52.31   running egg_info
#12 52.31   writing MulticoreTSNE.egg-info/PKG-INFO
#12 52.31   writing dependency_links to MulticoreTSNE.egg-info/dependency_links.txt
#12 52.31   writing requirements to MulticoreTSNE.egg-info/requires.txt
#12 52.31   writing top-level names to MulticoreTSNE.egg-info/top_level.txt
#12 52.31   reading manifest file 'MulticoreTSNE.egg-info/SOURCES.txt'
#12 52.31   reading manifest template 'MANIFEST.in'
#12 52.31   writing manifest file 'MulticoreTSNE.egg-info/SOURCES.txt'
#12 52.31   running build_ext
#12 52.31   cmake version 3.20.1
#12 52.31
#12 52.31   CMake suite maintained and supported by Kitware (kitware.com/cmake).
#12 52.31   CMake Error: Unknown argument --
#12 52.31   CMake Error: Run 'cmake --help' for all supported options.
#12 52.31
#12 52.31   ERROR: Cannot generate Makefile. See above errors.
#12 52.31   ----------------------------------------
#12 52.31   ERROR: Failed building wheel for MulticoreTSNE
#12 52.31     ERROR: Command errored out with exit status 1:
#12 52.31      command: /opt/conda/envs/ethos/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-5jx__mkf/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/ethos/include/python3.9/MulticoreTSNE
#12 52.31          cwd: /tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/
#12 52.31     Complete output (26 lines):
#12 52.31     running install
#12 52.31     running build
#12 52.31     running build_py
#12 52.31     creating build
#12 52.31     creating build/lib.linux-x86_64-3.9
#12 52.31     creating build/lib.linux-x86_64-3.9/MulticoreTSNE
#12 52.31     copying MulticoreTSNE/__init__.py -> build/lib.linux-x86_64-3.9/MulticoreTSNE
#12 52.31     creating build/lib.linux-x86_64-3.9/MulticoreTSNE/tests
#12 52.31     copying MulticoreTSNE/tests/test_base.py -> build/lib.linux-x86_64-3.9/MulticoreTSNE/tests
#12 52.31     copying MulticoreTSNE/tests/__init__.py -> build/lib.linux-x86_64-3.9/MulticoreTSNE/tests
#12 52.31     running egg_info
#12 52.31     writing MulticoreTSNE.egg-info/PKG-INFO
#12 52.31     writing dependency_links to MulticoreTSNE.egg-info/dependency_links.txt
#12 52.31     writing requirements to MulticoreTSNE.egg-info/requires.txt
#12 52.31     writing top-level names to MulticoreTSNE.egg-info/top_level.txt
#12 52.31     reading manifest file 'MulticoreTSNE.egg-info/SOURCES.txt'
#12 52.31     reading manifest template 'MANIFEST.in'
#12 52.31     writing manifest file 'MulticoreTSNE.egg-info/SOURCES.txt'
#12 52.31     running build_ext
#12 52.31     cmake version 3.20.1
#12 52.31
#12 52.31     CMake suite maintained and supported by Kitware (kitware.com/cmake).
#12 52.31     CMake Error: Unknown argument --
#12 52.31     CMake Error: Run 'cmake --help' for all supported options.
#12 52.31
#12 52.31     ERROR: Cannot generate Makefile. See above errors.
#12 52.31     ----------------------------------------
#12 52.31 ERROR: Command errored out with exit status 1: /opt/conda/envs/ethos/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/setup.py'"'"'; __file__='"'"'/tmp/pip-install-110qdwqv/multicoretsne_4b5d168de5e04f6c894a77b8595839b9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-5jx__mkf/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/envs/ethos/include/python3.9/MulticoreTSNE Check the logs for full command output.
#12 52.31
#12 52.31
#12 52.31 CondaEnvException: Pip failed
#12 52.31
------
executor failed running [/bin/bash -c conda-env create -n ${env_nm} -f ./${yaml_nm}     && conda clean --all --yes --force-pkgs-dirs     && find /opt/conda/ -follow -type f -name '*.a' -delete     && find /opt/conda/ -follow -type f -name '*.pyc' -delete     && find /opt/conda/ -follow -type f -name '*.js.map' -delete     && pip cache purge     && rm -rf /home/$NB_USER/.cache/pip     && rm ./${yaml_nm}]: exit code: 1
ndwarshuis commented 3 years ago

I just encountered the same error and can confirm downgrading cmake from 3.20.1 -> 3.19.7 was the issue. It seems the new version of cmake doesn't support the -- in its arguments. After downgrading, pip install multicoretsne works as expected.

This was effect was seen both in a pyenv/virtual environment where the python version was 3.9.2, and in a systemwide python installation where the python version was 3.9.3 (both on arch linux).

kmoad commented 3 years ago

Getting the same issue on Fedora 34 system install. Downgrading with dnf install cmake-3.19.7-1.fc34 resolved it, but it would be nice to be able to use latest cmake.

TheAustinator commented 3 years ago

pip install cmake==3.18.4 worked for me.

yash-goel commented 2 years ago

Hi, I still get the cmake error with 3.18.4 version.

ERROR: Command errored out with exit status 1:
   command: /home/yash/anaconda3/envs/calvin_try/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4_pnlmiw/multicoretsne_f755883637e7456cac20fedb190b6da2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4_pnlmiw/multicoretsne_f755883637e7456cac20fedb190b6da2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-bgyvtm0q
       cwd: /tmp/pip-install-4_pnlmiw/multicoretsne_f755883637e7456cac20fedb190b6da2/
  Complete output (29 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/MulticoreTSNE
  copying MulticoreTSNE/__init__.py -> build/lib.linux-x86_64-3.8/MulticoreTSNE
  creating build/lib.linux-x86_64-3.8/MulticoreTSNE/tests
  copying MulticoreTSNE/tests/test_base.py -> build/lib.linux-x86_64-3.8/MulticoreTSNE/tests
  copying MulticoreTSNE/tests/__init__.py -> build/lib.linux-x86_64-3.8/MulticoreTSNE/tests
  running egg_info
  writing MulticoreTSNE.egg-info/PKG-INFO
  writing dependency_links to MulticoreTSNE.egg-info/dependency_links.txt
  writing requirements to MulticoreTSNE.egg-info/requires.txt
  writing top-level names to MulticoreTSNE.egg-info/top_level.txt
  reading manifest file 'MulticoreTSNE.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE.txt'
  writing manifest file 'MulticoreTSNE.egg-info/SOURCES.txt'
  running build_ext
  cmake version 3.18.4

  CMake suite maintained and supported by Kitware (kitware.com/cmake).
  CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-install-4_pnlmiw/multicoretsne_f755883637e7456cac20fedb190b6da2/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".

  ERROR: Cannot generate Makefile. See above errors.
  ----------------------------------------
  ERROR: Failed building wheel for MulticoreTSNE
scottgigante-immunai commented 1 year ago

Solved with pip install git+https://github.com/jorvis/Multicore-TSNE. @DmitryUlyanov please merge #92 !

divyagupta25 commented 1 year ago

sudo apt-get install make worked for me

iTomxy commented 12 months ago

(Sept. 22, 2023) I follow the manual installing method in README, i.e.

git clone https://github.com/DmitryUlyanov/Multicore-TSNE.git
cd Multicore-TSNE/
pip install .    # <- got cmake error

but still got cmake error. My cmake version is 3.20.2. As a workaround, I manually modify the setup.py line 55, i.e. change

if version.parse(self.get_cmake_version()) < version.parse("3.22.0"):
    cmake_passthru_flag = "--"
else:
    cmake_passthru_flag = "-S"

to

# if version.parse(self.get_cmake_version()) < version.parse("3.22.0"):
#     cmake_passthru_flag = "--"
# else:
cmake_passthru_flag = "-S"

Then it install correctly.

CamiloMartinezM commented 6 months ago

Worked for me on a conda environment and cmake version 3.27.7.

> git clone https://github.com/DmitryUlyanov/Multicore-TSNE.git
Clonando en 'Multicore-TSNE'...
remote: Enumerating objects: 558, done.
remote: Counting objects: 100% (53/53), done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 558 (delta 21), reused 35 (delta 13), pack-reused 505
Recibiendo objetos: 100% (558/558), 456.25 KiB | 5.24 MiB/s, listo.
Resolviendo deltas: 100% (314/314), listo.
> cd Multicore-TSNE/
> conda activate multicoretsne
> pip install .
Processing /home/Camilo/Documentos/Repositorios/Multicore-TSNE
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /home/Camilo/anaconda3/envs/multicoretsne/lib/python3.11/site-packages (from MulticoreTSNE==0.1) (1.26.4)
Requirement already satisfied: cffi in /home/Camilo/anaconda3/envs/multicoretsne/lib/python3.11/site-packages (from MulticoreTSNE==0.1) (1.16.0)
Requirement already satisfied: pycparser in /home/Camilo/anaconda3/envs/multicoretsne/lib/python3.11/site-packages (from cffi->MulticoreTSNE==0.1) (2.21)
Building wheels for collected packages: MulticoreTSNE
  Building wheel for MulticoreTSNE (setup.py) ... done
  Created wheel for MulticoreTSNE: filename=MulticoreTSNE-0.1-cp311-cp311-linux_x86_64.whl size=71819 sha256=a6ba096a2a986753b207a0ddaafb90f1d4e4f5d7d6e0539123b06403369c728c
  Stored in directory: /home/Camilo/.cache/pip/wheels/f2/27/3a/e0b555687057204b7b11ff19c795409d809d348575b1e2eb43
Successfully built MulticoreTSNE
Installing collected packages: MulticoreTSNE
Successfully installed MulticoreTSNE-0.1