Python 3.11 support?

[sjaenick]

Hi,

compilation currently fails on python 3.11 as longintrepr.h no longer exists; could you take a look at this?

[...]
      running egg_info
      writing umi_tools.egg-info/PKG-INFO
      writing dependency_links to umi_tools.egg-info/dependency_links.txt
      writing entry points to umi_tools.egg-info/entry_points.txt
      writing requirements to umi_tools.egg-info/requires.txt
      writing top-level names to umi_tools.egg-info/top_level.txt
      reading manifest file 'umi_tools.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'umi_tools.egg-info/SOURCES.txt'
      copying umi_tools/_dedup_umi.c -> build/lib.linux-x86_64-cpython-311/umi_tools
      copying umi_tools/_dedup_umi.pyx -> build/lib.linux-x86_64-cpython-311/umi_tools
      running build_ext
      building 'umi_tools._dedup_umi' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/umi_tools
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.11 -c umi_tools/_dedup_umi.c -o build/temp.linux-x86_64-cpython-311/umi_tools/_dedup_umi.o
      umi_tools/_dedup_umi.c:196:12: fatal error: longintrepr.h: No such file or directory
        196 |   #include "longintrepr.h"
            |            ^~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for umi_tools

[TomSmithCGAT]

It looks like this should have been resolved in cython: https://github.com/cython/cython/pull/4428.

What version of cython are you using? Can you please try updating it and re-running.

[sjaenick]

https://github.com/CGATOxford/UMI-tools/blob/master/umi_tools/_dedup_umi.c needs to be recreated with a more recent cython version; afterwards, compilation succeeds.

[TomSmithCGAT]

Ah, of course. Thank you! I'll leave this issue open until this has been covered in the next release.

[Poshi]

Just hit the same issue. Hope the next release comes out soon!

[TomSmithCGAT]

Ack, forgot to tie this in to the recent release!

[TomSmithCGAT]

I've recompiled on the ts_recompile_dedup_umi.c branch and tests pass on py 3.7-3.9.

@Poshi and @sjaenick, could one of you please test whether this resolves the issue with py 3.11 please? I tried to set up an environment to test and hit some snags installing pysam that I don't have time to work through right now.

[sjaenick]

sj@azathoth:~$ python --version
Python 3.11.2
sj@azathoth:~$ git clone -b ts_recompile_dedup_umi.c https://github.com/CGATOxford/UMI-tools.git
Cloning into 'UMI-tools'...
remote: Enumerating objects: 4339, done.
remote: Counting objects: 100% (367/367), done.
remote: Compressing objects: 100% (151/151), done.
remote: Total 4339 (delta 208), reused 302 (delta 172), pack-reused 3972
Receiving objects: 100% (4339/4339), 27.77 MiB | 12.16 MiB/s, done.
Resolving deltas: 100% (3070/3070), done.
sj@azathoth:~$ cd UMI-tools/
sj@azathoth:~/UMI-tools$ python setup.py install
/home/sj/UMI-tools/setup.py:15: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(setuptools.__version__) < LooseVersion('1.1'):
/vol/mgx-sw/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running install
/vol/mgx-sw/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/vol/mgx-sw/lib/python3.11/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
creating umi_tools.egg-info
writing umi_tools.egg-info/PKG-INFO
writing dependency_links to umi_tools.egg-info/dependency_links.txt
writing entry points to umi_tools.egg-info/entry_points.txt
writing requirements to umi_tools.egg-info/requires.txt
writing top-level names to umi_tools.egg-info/top_level.txt
writing manifest file 'umi_tools.egg-info/SOURCES.txt'
reading manifest file 'umi_tools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'umi_tools.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/network.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/Documentation.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/extract_methods.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/whitelist.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/whitelist_methods.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/dedup.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/__init__.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/extract.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/Utilities.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/count_tab.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/version.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/umi_tools.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/umi_methods.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/sam_methods.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/count.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/group.py -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/_dedup_umi.c -> build/lib.linux-x86_64-cpython-311/umi_tools
copying umi_tools/_dedup_umi.pyx -> build/lib.linux-x86_64-cpython-311/umi_tools
running build_ext
building 'umi_tools._dedup_umi' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/umi_tools
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/vol/mgx-sw/include/python3.11 -c umi_tools/_dedup_umi.c -o build/temp.linux-x86_64-cpython-311/umi_tools/_dedup_umi.o
umi_tools/_dedup_umi.c: In function ‘__Pyx_AddTraceback’:
umi_tools/_dedup_umi.c:438:62: error: dereferencing pointer to incomplete type ‘PyFrameObject’ {aka ‘struct _frame’}
  438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
      |                                                              ^~
umi_tools/_dedup_umi.c:2408:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
 2408 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/gcc' failed with exit code 1
sj@azathoth:~/UMI-tools$ 

[TomSmithCGAT]

Somehow I managed to compile with cython 0.29.28 not >=0.29.29 as intended 🤦. Could you please pull and retry.

[sjaenick]

Done... it's working now.

[TomSmithCGAT]

Lovely. Thanks @sjaenick 👍

[Poshi]

Oh! I'm late to the party... thanks for the effort! I will check ASAP and get back if I find any issue :-)