Compilation fails on python 3.12

[s-andrews]

Using AlmaLinux 9.2 (RHEL9 derivative) with gcc 11.4.1 and python 3.12.2

Compilation using pip3 install umi-tools fails with:

      umi_tools/_dedup_umi.c: In function ‘__pyx_f_9umi_tools_10_dedup_umi_edit_distance’:
      umi_tools/_dedup_umi.c:1322:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       1322 |     __Pyx_GetModuleGlobalName(__pyx_t_4, __pyx_n_s_np); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 14, __pyx_L1_error)
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_get_tp_dict_version’:
      umi_tools/_dedup_umi.c:1931:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       1931 |     return likely(dict) ? __PYX_GET_DICT_VERSION(dict) : 0;
            |     ^~~~~~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_get_object_dict_version’:
      umi_tools/_dedup_umi.c:1943:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       1943 |     return (dictptr && *dictptr) ? __PYX_GET_DICT_VERSION(*dictptr) : 0;
            |     ^~~~~~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_object_dict_version_matches’:
      umi_tools/_dedup_umi.c:1947:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       1947 |     if (unlikely(!dict) || unlikely(tp_dict_version != __PYX_GET_DICT_VERSION(dict)))
            |     ^~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c: In function ‘__Pyx__GetModuleGlobalName’:
      umi_tools/_dedup_umi.c:1964:5: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       1964 |     __PYX_UPDATE_DICT_CACHE(__pyx_d, result, *dict_cached_value, *dict_version)
            |     ^~~~~~~~~~~~~~~~~~~~~~~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_CLineForTraceback’:
      umi_tools/_dedup_umi.c:2274:9: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       2274 |         __PYX_PY_DICT_LOOKUP_IF_MODIFIED(
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c:2274:9: warning: ‘ma_version_tag’ is deprecated [-Wdeprecated-declarations]
       2274 |         __PYX_PY_DICT_LOOKUP_IF_MODIFIED(
            |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from /bi/apps/python/3.12.2/include/python3.12/dictobject.h:90,
                       from /bi/apps/python/3.12.2/include/python3.12/Python.h:61,
                       from umi_tools/_dedup_umi.c:6:
      /bi/apps/python/3.12.2/include/python3.12/cpython/dictobject.h:22:34: note: declared here
         22 |     Py_DEPRECATED(3.12) uint64_t ma_version_tag;
            |                                  ^~~~~~~~~~~~~~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_PyInt_As_int’:
      umi_tools/_dedup_umi.c:2537:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
       2537 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      umi_tools/_dedup_umi.c:2592:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
       2592 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_PyInt_As_long’:
      umi_tools/_dedup_umi.c:2809:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
       2809 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      umi_tools/_dedup_umi.c:2864:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
       2864 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
            |                                                     ^~
      umi_tools/_dedup_umi.c: In function ‘__Pyx_PyIndex_AsSsize_t’:
      umi_tools/_dedup_umi.c:3315:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
       3315 |     const digit* digits = ((PyLongObject*)b)->ob_digit;
            |                                             ^~
      error: command '/usr/bin/gcc' failed with exit code 1

It looks like it might be similar to this issue raised against cython: https://github.com/cython/cython/issues/5238

Doing the same thing with python 3.11.8 on the same system works as expected.

[sebastian-luna-valero]

Could you please try https://umi-tools.readthedocs.io/en/latest/INSTALL.html#conda-package-manager

[s-andrews]

It will almost certainly work with conda- it works if I use pip from python v3.11 on the same machine. The failure seems to be related to a change in internals in python 3.12. I have it working now, but thought I'd report it in case you had a fix for future 3.12 users.

[TomSmithCGAT]

@s-andrews - Thanks for bringing this to our attention.

Forgive my ignorance, but is the solution simply that we need to re-compile umi_tools/_dedup_umi.c with a more recent cython? Will that be backwards compatible?

[mmokrejs]

I am getting the same with umi_tools-1.1.4 under python-3.12. The following works for me around the bug:

python3.11 -m pip install umi_tools

[mmokrejs]

Could you please try https://umi-tools.readthedocs.io/en/latest/INSTALL.html#conda-package-manager

The package manager did not resolve deps for me so it did not even try to install the package. I had to stick to pip.

[bounlu]

umi_tools cannot be installed with python 3.12 under conda:

$ conda install "umi_tools>=1.1.5"
Channels:
 - conda-canary
 - bioconda
 - r
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: - warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package umi_tools-1.1.5-py310h4b81fae_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ pin-1 is installable and it requires
│  └─ python 3.12.* , which can be installed;
└─ umi_tools >=1.1.5  is not installable because there are no viable options
   ├─ umi_tools 1.1.5 would require
   │  └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
   ├─ umi_tools 1.1.5 would require
   │  └─ python >=3.11,<3.12.0a0 , which conflicts with any installable versions previously reported;
   ├─ umi_tools 1.1.5 would require
   │  └─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported;
   └─ umi_tools 1.1.5 would require
      └─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported.

Pins seem to be involved in the conflict. Currently pinned specs:
 - python 3.12.* (labeled as 'pin-1')

[sebastian-luna-valero]

Currently there seems to be an issue building this package for python 3.12.

With conda, you should be able to create a dedicated environment with a different version of python (e.g. 3.11) for umi_tools:

conda create -n umi_tools -c bioconda -c conda-forge umi_tools
conda activate umi_tools
umi_tools --help
python --version

[IanSudbery]

As of #657, umi_tools should now compile on python 3.12.