Python 3.11 compatibility

FriedrichFroebel commented 2 years ago

I just started evaluating whether gamera-4 already works with the latest release candidate of Python 3.11. Doing so, I performed some initial clean-up of build warnings as well to reduce the build noise, as I still did not manage to get a successful package. My changes are currently available at https://github.com/FriedrichFroebel/gamera-4.

What I did so far:

Py_TYPE had to be changed to Py_SET_TYPE.
The VIGRA build warnings have been fixed by using the upstream replacements of the offending sections.
Some eodev warnings concerning wrongly indented code have been fixed.

What I might fix in the near future to further reduce the build noise:

The buffer API changes for Python.
The call API changes for Python.

While Python 3.8 to Python 3.10 still work with these changes, I am facing failing tests on Python 3.11:

============================= test session starts ==============================
platform linux -- Python 3.11.0rc2, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/runner/work/gamera-4/gamera-4
collected 285 items / 16 errors

==================================== ERRORS ====================================
__________________ ERROR collecting tests/test_classifier.py ___________________
tests/test_classifier.py:1: in <module>
    from gamera import knn, classify, gamera_xml
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/knn.py:23: in <module>
    from gamera import core, util, config, classify
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
_____________________ ERROR collecting tests/test_color.py _____________________
tests/test_color.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
___________________ ERROR collecting tests/test_features.py ____________________
tests/test_features.py:1: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
__________________ ERROR collecting tests/test_floatpoint.py ___________________
tests/test_floatpoint.py:3: in <module>
    from gamera.gameracore import FloatPoint as FP
E   SystemError: initialization of gameracore raised unreported exception
________________ ERROR collecting tests/test_graph_colorize.py _________________
tests/test_graph_colorize.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
_____________________ ERROR collecting tests/test_image.py _____________________
tests/test_image.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
__________________ ERROR collecting tests/test_image_info.py ___________________
tests/test_image_info.py:1: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
__________________ ERROR collecting tests/test_load_image.py ___________________
tests/test_load_image.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
_____________________ ERROR collecting tests/test_mlcc.py ______________________
tests/test_mlcc.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
__________________ ERROR collecting tests/test_nested_list.py __________________
tests/test_nested_list.py:1: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
____________________ ERROR collecting tests/test_plugins.py ____________________
tests/test_plugins.py:1: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
_____________________ ERROR collecting tests/test_rect.py ______________________
tests/test_rect.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
______________________ ERROR collecting tests/test_rle.py ______________________
tests/test_rle.py:1: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
__________________ ERROR collecting tests/test_save_image.py ___________________
tests/test_save_image.py:1: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
____________________ ERROR collecting tests/test_voronoi.py ____________________
tests/test_voronoi.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
______________________ ERROR collecting tests/test_xml.py ______________________
tests/test_xml.py:3: in <module>
    from gamera.core import *
/opt/hostedtoolcache/Python/3.11.0-rc.2/x64/lib/python3.11/site-packages/gamera-4.0.0-py3.11-linux-x86_64.egg/gamera/core.py:51: in <module>
    from gamera.gameracore import UNCLASSIFIED, AUTOMATIC, HEURISTIC, MANUAL
E   SystemError: initialization of gameracore raised unreported exception
=========================== short test summary info ============================
ERROR tests/test_classifier.py - SystemError: initialization of gameracore ra...
ERROR tests/test_color.py - SystemError: initialization of gameracore raised ...
ERROR tests/test_features.py - SystemError: initialization of gameracore rais...
ERROR tests/test_floatpoint.py - SystemError: initialization of gameracore ra...
ERROR tests/test_graph_colorize.py - SystemError: initialization of gameracor...
ERROR tests/test_image.py - SystemError: initialization of gameracore raised ...
ERROR tests/test_image_info.py - SystemError: initialization of gameracore ra...
ERROR tests/test_load_image.py - SystemError: initialization of gameracore ra...
ERROR tests/test_mlcc.py - SystemError: initialization of gameracore raised u...
ERROR tests/test_nested_list.py - SystemError: initialization of gameracore r...
ERROR tests/test_plugins.py - SystemError: initialization of gameracore raise...
ERROR tests/test_rect.py - SystemError: initialization of gameracore raised u...
ERROR tests/test_rle.py - SystemError: initialization of gameracore raised un...
ERROR tests/test_save_image.py - SystemError: initialization of gameracore ra...
ERROR tests/test_voronoi.py - SystemError: initialization of gameracore raise...
ERROR tests/test_xml.py - SystemError: initialization of gameracore raised un...
!!!!!!!!!!!!!!!!!!! Interrupted: 16 errors during collection !!!!!!!!!!!!!!!!!!!
============================== 16 errors in 0.97s ==============================

I am not sure what the actual issue is here, as the message is not really helpful. Only test_graph.py and test_kdtree.py are not affected here. The build seems to work correctly (at least there are no errors any more, just warnings). Does anyone have an idea about this and how it could be fixed?

FriedrichFroebel commented 2 years ago

If someone wants to test it or have a look at the generated wheels: https://github.com/FriedrichFroebel/gamera-4/actions/workflows/wheels.yml. If the downloads are missing (retention days have been reached), just drop me a comment and I will re-generate them. By default, wheels for Python 3.10 and Python 3.11 are provided as assets.

cdalitz commented 2 years ago

Dear Friedrich, thanks for the contributions which I have just merged into the main branch. I would like to add your name to the ACKNOWLEDGEMENTS file, but do not know whether your name is correctly written with "oe" or with an umlaut. Would you mind letting me know? Thanks, Christoph

FriedrichFroebel commented 2 years ago

@cdalitz Just to clarify: FriedrichFröbel is just the nick name I use on GitHub (and at some other places) - I prefer to not disclose my real name here. If you want to add me to the ACKNOWLEDGEMENTS, just use FriedrichFröbel or FriedrichFroebel, where I prefer the first one (the second one just avoids the umlauts to simplify stuff for non-German sites/users).

cdalitz commented 2 years ago

Done. Maybe we will have one day more contributors with celebrities as nicknames, which makes the "hall of fame" very illustruos ;-)

FriedrichFroebel commented 2 years ago

To come back to this issue: I had some look at this issue again, but it still remains unclear to me what actually causes this.

What I tested: Built Python 3.11.0rc2 on an Ubuntu 22.04 machine and tested both the CI-generated wheel and a local package build. When running

import gamera

print(gamera.__version__)
print(dir(gamera))

from gamera import core

there, everything will run fine - except for the last import, where we will get the aforementioned stack trace. For now, I did not manage to get something helpful out of GDB which would help here. From what I have read, this error could be caused by any build or linking issue.

FriedrichFroebel commented 1 year ago

While there still is an issue with building from #61, it seems like it fixes Python 3.11 support as well (at least with my custom Python 3.11.0rc2 build previously used locally as well and the above code).

Build error:

building 'gamera.plugins._fourier_features' extension
[...]
cc1plus: fatal error: src/geostructs/delaunaytree.cpp: Datei oder Verzeichnis nicht gefunden
compilation terminated.

cdalitz commented 1 year ago

This should be fixed with the latest commit. I have only tested it with python 3.5. Can you please test it with python 3.11, too? Thanks.

FriedrichFroebel commented 1 year ago

As mentioned in the above comment, Python 3.11 seems to work now. This is backed through the gamera-4 CI using Python 3.11 as well, too.

hsnr-gamera / gamera-4

Python 3.11 compatibility #54