installation issue

[haircell]

Hi: I'm having some trouble installing palantir and would appreciate some guidance please.

After pip install palantir or pip3 install palantir in my virtual environment, I get the following error. Running Python 3.8.5 on a Mac M1

Thank you

Collecting palantir
  Obtaining dependency information for palantir from https://files.pythonhosted.org/packages/9e/63/9198e3f48d0acb0a66643a4b824cf3714bfd22d78c1c2e27b8f2ffe7bfda/palantir-1.3.0-py3-none-any.whl.metadata
  Using cached palantir-1.3.0-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: numpy>=1.14.2 in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (1.23.5)
Requirement already satisfied: pandas>=0.22.0 in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (1.5.3)
Requirement already satisfied: scipy>=1.3 in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (1.10.1)
Requirement already satisfied: networkx>=2.1 in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (2.8.8)
Requirement already satisfied: scikit-learn in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (1.2.1)
Requirement already satisfied: joblib in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (1.2.0)
Collecting fcsparser>=0.1.2 (from palantir)
  Obtaining dependency information for fcsparser>=0.1.2 from https://files.pythonhosted.org/packages/f0/29/6bba9b5003b3f2a174cad0cd9286fd7babf1e512b4e450f7d84f23316d06/fcsparser-0.2.7-py3-none-any.whl.metadata
  Using cached fcsparser-0.2.7-py3-none-any.whl.metadata (3.6 kB)
Requirement already satisfied: leidenalg>=0.9.1 in /Users/hcuser/miniconda/lib/python3.10/site-packages (from palantir) (0.9.1)
Collecting tables>=3.4.2 (from palantir)
  Using cached tables-3.8.0.tar.gz (8.0 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      <string>:19: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      ld: library not found for -llzo2
      clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
      ld: library not found for -llzo
      clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
      ld: library not found for -lblosc
      clang-15: error: linker command failed with exit code 1 (use -v to see invocation)

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

      def _dump_h5_backtrace():
          cdef object bt = []

          if H5Ewalk(H5E_DEFAULT, H5E_WALK_DOWNWARD, e_walk_cb, <void*>bt) < 0:
                                                     ^
      ------------------------------------------------------------

      tables/utilsextension.pyx:384:47: Cannot assign type 'herr_t (unsigned int, const H5E_error_t *, void *) except? -1 nogil' to 'H5E_walk_t'. Exception values are incompatible. Suggest adding 'noexcept' to type 'herr_t (unsigned int, const H5E_error_t *, void *) except? -1 nogil'.
      cpuinfo failed, assuming no CPU features: 'flags'
      * Using Python 3.10.9 (main, Jan 11 2023, 09:18:18) [Clang 14.0.6 ]
      * Found cython 3.0.2
      * USE_PKGCONFIG: True
      * Found conda env: ``/Users/hcuser/miniconda``
      * Found HDF5 headers at ``/Users/hcuser/miniconda/include``, library at ``/Users/hcuser/miniconda/lib``.
      * Could not find LZO 2 headers and library; disabling support for it.
      * Could not find LZO 1 headers and library; disabling support for it.
      * Found bzip2 headers at ``/Users/hcuser/miniconda/include``, library at ``/Users/hcuser/miniconda/lib``.
      * Could not find blosc headers and library; using internal sources.
      * Found blosc2 headers at ``/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/include``, library at ``/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/lib``.
        * Copying blosc2 runtime library to 'tables' dir because it was not found in standard locations
      Compiling tables/utilsextension.pyx because it changed.
      [1/1] Cythonizing tables/utilsextension.pyx
      Traceback (most recent call last):
        File "/Users/hcuser/miniconda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/hcuser/miniconda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/hcuser/miniconda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 928, in <module>
        File "<string>", line 923, in get_cython_extfiles
        File "/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
          cythonize_one(*args)
        File "/private/var/folders/yf/w7thwhb573n49lkkqm01nf980000gp/T/pip-build-env-owfkruls/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
          raise CompileError(None, pyx_file)
      Cython.Compiler.Errors.CompileError: tables/utilsextension.pyx
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[katosh]

Hi @haircell,

I understand you're having issues installing Palantir on a Mac with an M1 chip. You're not alone; this seems to be a known issue when installing pytables on M1 Macs. Specifically, it often happens because non-python dependencies are not managed by pip. This is further discussed in this Stack Overflow thread.

Why Conda Over pip?

conda is generally more effective than pip for managing complex dependencies and binaries, especially on non-standard architectures like the Apple M1. It can manage libraries that are not Python libraries and ensure that everything is compatible. This is why I would recommend it to solve the issue you described.

Solution Using Conda (Recommended)

1. First, install Miniconda or Anaconda if you haven't already. Download it from here.

2. Create a new conda environment and activate it:

   conda create -n myenv python=3.8 # generally I would recommend a newer Python here
   conda activate myenv

3. Now, install PyTables and any other problematic dependencies:

   conda install pytables hdf5 c-blosc lzo bzip2

4. Finally, try installing Palantir again:

   pip install palantir

Manual Build (Alternative)

If you prefer not to use conda, you can manually install the dependencies:

1. Install Homebrew for Mac M1:

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Install required packages using Homebrew:

   brew install hdf5 c-blosc lzo bzip2

3. Then install Palantir:

   pip install palantir

I hope this helps resolve the issue you're facing with installing Palantir on your Mac M1. Feel free to reach out if you have more questions.

[katosh]

Hi @haircell,

I've identified that the tables dependency, which was causing the installation issue on Mac M1 systems, is actually not in use within Palantir's current codebase. To address this, I've created a patch that removes it.

You can install the patched version using the following command:

pip install 'git+https://github.com/settylab/Palantir.git'

[haircell]

Thank you for the detailed first note, and the amazing patch! Worked like a charm!

[GabyBG]

Just sharing in case someone is having the same problem as me. I am using an EC2 instance with GPU processing.

I was having a different issue, that also got solved by using the patch (https://github.com/dpeerlab/Palantir/issues/122#issuecomment-1740128640) after running

conda install pytables hdf5 c-blosc lzo bzip2

This was my error

Collecting palantir
  Obtaining dependency information for palantir from https://files.pythonhosted.org/packages/9e/63/9198e3f48d0acb0a66643a4b824cf3714bfd22d78c1c2e27b8f2ffe7bfda/palantir-1.3.0-py3-none-any.whl.metadata
  Using cached palantir-1.3.0-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: numpy>=1.14.2 in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (1.24.4)
Requirement already satisfied: pandas>=0.22.0 in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (1.3.5)
Requirement already satisfied: scipy>=1.3 in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (1.10.1)
Requirement already satisfied: networkx>=2.1 in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (2.8.8)
Requirement already satisfied: scikit-learn in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (1.3.1)
Requirement already satisfied: joblib in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (1.3.2)
Collecting fcsparser>=0.1.2 (from palantir)
  Obtaining dependency information for fcsparser>=0.1.2 from https://files.pythonhosted.org/packages/f0/29/6bba9b5003b3f2a174cad0cd9286fd7babf1e512b4e450f7d84f23316d06/fcsparser-0.2.7-py3-none-any.whl.metadata
  Using cached fcsparser-0.2.7-py3-none-any.whl.metadata (3.6 kB)
Requirement already satisfied: leidenalg>=0.9.1 in ./anaconda3/envs/scFates-gpu/lib/python3.8/site-packages (from palantir) (0.10.1)
Collecting tables>=3.4.2 (from palantir)
  Using cached tables-3.9.0.tar.gz (4.7 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Collecting setuptools>=61.0.0
        Obtaining dependency information for setuptools>=61.0.0 from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
        Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
      Collecting wheel
        Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
        Using cached wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
      Collecting oldest-supported-numpy
        Obtaining dependency information for oldest-supported-numpy from https://files.pythonhosted.org/packages/94/9a/756fef9346e5ca2289cb70d73990b4c9f25446a885c1186cfb93a85e7da0/oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata
        Using cached oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata (9.5 kB)
      Collecting packaging
        Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl.metadata
        Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
      Collecting py-cpuinfo
        Using cached py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)
      Collecting Cython>=0.29.32
        Obtaining dependency information for Cython>=0.29.32 from https://files.pythonhosted.org/packages/e1/c7/35f368d0e9140939df898e3befb2931bbe6d7b0ca69598b0ac20ee38e978/Cython-3.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
        Using cached Cython-3.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
      ERROR: Ignored the following versions that require a different python version: 2.2.8 Requires-Python <4,>=3.9
      ERROR: Could not find a version that satisfies the requirement blosc2>=2.2.8 (from versions: 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10, 0.2.0, 0.3.0, 0.3.1, 0.3.2, 0.4.0, 0.4.1, 0.5.1, 0.5.2, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 2.0.0, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.2.6, 2.2.7)
      ERROR: No matching distribution found for blosc2>=2.2.8
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

Looking forward to use palantir on my datasets

[katosh]

The patched version is now released as Palantir v1.3.1 and can be installed with

pip install --upgrade palantir

Thank you for the feedback!