Package not found on conda

[semvijverberg]

Dear Jasper,

I tried to install the package via conda and pip, and neither options seem to work.

mamba create --name test -c conda-forge sklearn-quantile

gives:

Looking for: ['sklearn-quantile']

warning  libmamba Cache file "/Users/semv/miniconda3/pkgs/cache/a850f475.json" was modified by another program
warning  libmamba Cache file "/Users/semv/miniconda3/pkgs/cache/09cdf8bf.json" was modified by another program
warning  libmamba Cache file "/Users/semv/miniconda3/pkgs/cache/9e99ffaf.json" was modified by another program
warning  libmamba Cache file "/Users/semv/miniconda3/pkgs/cache/3e39a7aa.json" was modified by another program
warning  libmamba Cache file "/Users/semv/miniconda3/pkgs/cache/8bd55712.json" was modified by another program
warning  libmamba Cache file "/Users/semv/miniconda3/pkgs/cache/4ea078d6.json" was modified by another program
pkgs/r/osx-arm64                                   118.0 B @   1.6kB/s  0.1s
pkgs/main/noarch                                   852.1kB @   1.8MB/s  0.5s
pkgs/r/noarch                                        1.3MB @   1.8MB/s  0.7s
pkgs/main/osx-arm64                                  1.9MB @   2.4MB/s  0.8s
[+] 1.3s
conda-forge/osx-arm64                                7.3MB @   3.8MB/s  2.1s
conda-forge/noarch                                  13.9MB @   3.6MB/s  4.2s
Could not solve for environment specs
The following package could not be installed
└─ sklearn-quantile does not exist (perhaps a typo or a missing channel).

Installing it via pip gives the error that Cython was not install, after explicitly installing it, it give the error that numpy was not installed. (I was thinking it was not correctly resolving the dependencies.) After installing numpy, it said:

(bwcore) ➜  ~ pip install sklearn-quantile
Collecting sklearn-quantile
  Using cached sklearn_quantile-0.0.21.tar.gz (33 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [288 lines of output]
      /Users/semv/miniconda3/envs/bwcore/lib/python3.10/site-packages/Cython/Compiler/Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /private/var/folders/69/l8scbc7s3z799nds71hbtw0r0000gn/T/pip-install-85o5dsqh/sklearn-quantile_3c7c97c814c74dbbbe846210cd5103c0/sklearn_quantile/ensemble/maximum.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      /Users/semv/miniconda3/envs/bwcore/lib/python3.10/site-packages/Cython/Compiler/Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /private/var/folders/69/l8scbc7s3z799nds71hbtw0r0000gn/T/pip-install-85o5dsqh/sklearn-quantile_3c7c97c814c74dbbbe846210cd5103c0/sklearn_quantile/ensemble/quantile.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      /Users/semv/miniconda3/envs/bwcore/lib/python3.10/site-packages/Cython/Compiler/Main.py:384: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /private/var/folders/69/l8scbc7s3z799nds71hbtw0r0000gn/T/pip-install-85o5dsqh/sklearn-quantile_3c7c97c814c74dbbbe846210cd5103c0/sklearn_quantile/utils/weighted_quantile.pxd
        tree = Parsing.p_module(s, pxd, full_module_name)

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          void qsort(void *base, int nmemb, int size,
                  int(*compar)(const_void *, const_void *)) nogil

      cdef struct IndexedElement:
          np.ulong_t index
          ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:20:4: 'ulong_t' is not a type identifier

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          if v >= 0: return 1

      cdef long[:] argsort(float[:] data) nogil:
          """source: https://github.com/jcrudy/cython-argsort/blob/master/cyargsort/argsort.pyx"""
          cdef np.ulong_t i
               ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:35:9: 'ulong_t' is not a type identifier

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

      cdef long[:] argsort(float[:] data) nogil:
          """source: https://github.com/jcrudy/cython-argsort/blob/master/cyargsort/argsort.pyx"""
          cdef np.ulong_t i
          cdef np.ulong_t n = data.shape[0]
               ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:36:9: 'ulong_t' is not a type identifier

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          # Allocate index tracking array.
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
              order_struct[i].index = i
                          ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:47:20: Invalid index type '<error>'

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
              order_struct[i].index = i
              order_struct[i].value = data[i]
                                           ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:48:37: Invalid index for memoryview specified, type <error>

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
              order_struct[i].index = i
              order_struct[i].value = data[i]
                          ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:48:20: Invalid index type '<error>'

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          for i in range(n):
              order_struct[i].index = i
              order_struct[i].value = data[i]

          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)
                                                                  ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:51:60: Cannot assign type 'int (const_void *, const_void *) except? -1' to 'int (*)(const_void *, const_void *) noexcept'. Exception values are incompatible. Suggest adding 'noexcept' to type 'int (const_void *, const_void *) except? -1'.

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
              order[i] = order_struct[i].index
                                     ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:55:31: Invalid index type '<error>'

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
              order[i] = order_struct[i].index
                    ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:55:14: Invalid index for memoryview specified, type <error>

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Allocate index tracking array.
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
                   ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:46:13: Coercion from Python not allowed without the GIL

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Allocate index tracking array.
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
                   ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:46:13: Iterating over Python object not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Allocate index tracking array.
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
                        ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:46:18: Calling gil-requiring function not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Allocate index tracking array.
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
                        ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:46:18: Constructing Python tuple not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Allocate index tracking array.
          cdef IndexedElement *order_struct = <IndexedElement *> malloc(n * sizeof(IndexedElement))

          # Copy data into index tracking array.
          for i in range(n):
                         ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:46:19: Converting to Python object not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
                   ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:54:13: Coercion from Python not allowed without the GIL

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
                   ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:54:13: Iterating over Python object not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
                        ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:54:18: Calling gil-requiring function not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
                        ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:54:18: Constructing Python tuple not allowed without gil

      Error compiling Cython file:
      ------------------------------------------------------------
      ...

          # Sort index tracking array.
          qsort(<void *> order_struct, n, sizeof(IndexedElement), _compare)

          # Copy indices from index tracking array to output array.
          for i in range(n):
                         ^
      ------------------------------------------------------------

      sklearn_quantile/utils/weighted_quantile.pyx:54:19: Converting to Python object not allowed without gil
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/69/l8scbc7s3z799nds71hbtw0r0000gn/T/pip-install-85o5dsqh/sklearn-quantile_3c7c97c814c74dbbbe846210cd5103c0/setup.py", line 30, in <module>
          ext_modules=cythonize(extensions),
        File "/Users/semv/miniconda3/envs/bwcore/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
          cythonize_one(*args)
        File "/Users/semv/miniconda3/envs/bwcore/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
          raise CompileError(None, pyx_file)
      Cython.Compiler.Errors.CompileError: sklearn_quantile/utils/weighted_quantile.pyx
      Compiling sklearn_quantile/utils/weighted_quantile.pyx because it changed.
      Compiling sklearn_quantile/ensemble/quantile.pyx because it changed.
      Compiling sklearn_quantile/ensemble/maximum.pyx because it changed.
      [1/3] Cythonizing sklearn_quantile/ensemble/maximum.pyx
      [2/3] Cythonizing sklearn_quantile/ensemble/quantile.pyx
      [3/3] Cythonizing sklearn_quantile/utils/weighted_quantile.pyx
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I hope this helps.

Here also some system information:

    System Software Overview:

      System Version: macOS 13.2.1 (22D68)
      Kernel Version: Darwin 22.3.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: Mac14,9
      Model Number: Z17L001PWN/A
      Chip: Apple M2 Pro
      Total Number of Cores: 12 (8 performance and 4 efficiency)
      Memory: 32 GB
      System Firmware Version: 8419.80.7
      OS Loader Version: 8419.80.7
      Serial Number (system): XFKGP4R25H
      Hardware UUID: 4FDD49BF-5D80-5338-9002-87C3BB74E3D7
      Provisioning UDID: 00006020-000470810A87401E
      Activation Lock Status: Enabled 

[jasperroebroek]

Dear Sem,

This seems very strange. I have run the same command on mamba, and works without problems. Running the same operating system. Maybe you could try to delete the mamba cache and try the operation again.

The saw the issue with pip not picking up on the dependencies. I have been trying to resolve this, but haven't managed yet. What if you try the following.

mamba create -n test pip
mamba activate test
pip install packaging numpy cython
pip install sklearn-quantile

This works for me and installs the latest version of sklearn-quantile (v0.0.21). Let me know if you manage to make this work.

Cheers, Jasper

[jasperroebroek]

Also, which cython version did you use for compilation when taking the pypi route?

[semvijverberg]

Dear Jasper,

Thanks for your help!

I copied and executed:

mamba create -n test pip
mamba activate test
pip install packaging numpy cython
pip install sklearn-quantile

Unfortunately I got the identical Cython error. The Cython version it installed is 3.0.2.

Maybe it relates to my Mac M2 chip / Cython compatibility? I will google a bit.

[semvijverberg]

FYI, I forked the repo and managed to install it when changing the datatypes from np.ulong_t to int. I have no experience with Cython code, so I was just wildly guessing here. However, newer versions of Cython gives an error, I also had to degrade the Cython version to 0.29.34.

cdef struct IndexedElement:
    int index
    np.float32_t value

cdef int _compare(const_void *a, const_void *b):
    cdef np.float32_t v
    if isnan((<IndexedElement*> a).value): return 1
    if isnan((<IndexedElement*> b).value): return -1
    v = (<IndexedElement*> a).value-(<IndexedElement*> b).value
    if v < 0: return -1
    if v >= 0: return 1

cdef long[:] argsort(float[:] data) nogil:
    """source: https://github.com/jcrudy/cython-argsort/blob/master/cyargsort/argsort.pyx"""
    cdef int i
    cdef int n = data.shape[0]
    cdef long[:] order

[semvijverberg]

My current solution:

conda create -n test pip
conda activate test
pip install packaging numpy cython==0.29.34
pip install "git+https://github.com/Beyond-Weather-Git/sklearn-quantile.git"

[semvijverberg]

I updated the setup.cfg to pyproject.toml (which allows to specify installations (Cython + numpy) that are needed for the build (installation of sklearn-quantile).

 pip install git+https://github.com/Beyond-Weather-Git/sklearn-quantile.git@updated_pyproject_toml

[jasperroebroek]

Thanks for the update. I have checked the numpy documentation and they suggest moving from np.ulong_t to np.intp_t. I just tested it with the new Cython version, and that compiled without issues. That seems much cleaner than depending on an old version of Cython.

I never used pyproject.toml. Do you think that is recognised by conda-forge?

[semvijverberg]

Dear Jasper,

Thanks for the update!

I believe they are currently working on pyproject support: https://github.com/conda/conda/issues/12462

[jasperroebroek]

I have updated the code and pushed everything to PyPI and conda (version 0.0.29). It should now again be installable without issues. I will close this issue, if it persists feel free to reopen.