imagecodecs 2024.9.22 issue with GIL

[Schamschula]

While attempting to update MacPorts imagecodecs to version 2024.9.22 I ran into an issue with GIL:

/usr/bin/clang -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -arch arm64 -arch arm64 -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk build/temp.macosx-14.0-arm64-cpython-312/3rdparty/openjpeg/color.o build/temp.macosx-14.0-arm64-cpython-312/imagecodecs/_jpeg2k.o -lm -lopenjp2 -llcms2 -o build/lib.macosx-14.0-arm64-cpython-312/imagecodecs/_jpeg2k.cpython-312-darwin.so

Error compiling Cython file:
------------------------------------------------------------
...
        err.pub.output_message = my_output_message

        if setjmp(err.setjmp_buffer):
            # msg = err.pub.jpeg_message_table[err.pub.msg_code]
            msg[0] = b'\x00'
            err.pub.format_message(<jpeg_common_struct*> &cinfo, &msg[0])
                                  ^
------------------------------------------------------------

imagecodecs/_jpeg8_legacy.pyx:215:34: Calling gil-requiring function not allowed without gil

Error compiling Cython file:
------------------------------------------------------------
...
        err.pub.error_exit = my_error_exit
        err.pub.output_message = my_output_message
        if setjmp(err.setjmp_buffer):
            # msg = err.pub.jpeg_message_table[err.pub.msg_code]
            msg[0] = b'\x00'
            err.pub.format_message(<jpeg_common_struct*> &cinfo, &msg[0])
                                  ^
------------------------------------------------------------

imagecodecs/_jpeg8_legacy.pyx:332:34: Calling gil-requiring function not allowed without gil
Compiling imagecodecs/_jpeg8_legacy.pyx because it changed.
[1/1] Cythonizing imagecodecs/_jpeg8_legacy.pyx
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
    main()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 271, in build_wheel
    return _build_backend().build_wheel(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/build_meta.py", line 421, in build_wheel
    return self._build_with_temp_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
    self.run_setup()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 738, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 183, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
    dist.run_commands()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/command/bdist_wheel.py", line 398, in run
    self.run_command("build")
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 98, in run
    _build_ext.run(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
    self.build_extensions()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
    self._build_extensions_serial()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
    self.build_extension(ext)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 263, in build_extension
    _build_ext.build_extension(self, ext)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
    new_ext = cythonize(
              ^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
    cythonize_one(*args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: imagecodecs/_jpeg8_legacy.pyx

ERROR Backend subprocess exited when trying to invoke build_wheel

[cgohlke]

Can you try to build with:

diff --git a/imagecodecs/libjpeg.pxd b/imagecodecs/libjpeg.pxd
index 501d468..45af300 100644
--- a/imagecodecs/libjpeg.pxd
+++ b/imagecodecs/libjpeg.pxd
@@ -47,9 +47,9 @@ cdef extern from 'jpeglib.h' nogil:
     struct jpeg_error_mgr:
         int msg_code
         const char** jpeg_message_table
-        noreturn_t error_exit(jpeg_common_struct*)
-        void output_message(jpeg_common_struct*)
-        void format_message(jpeg_common_struct* cinfo, char* buffer)
+        noreturn_t error_exit(jpeg_common_struct*) nogil
+        void output_message(jpeg_common_struct*) nogil
+        void format_message(jpeg_common_struct* cinfo, char* buffer) nogil

     struct jpeg_common_struct:
         jpeg_error_mgr* err

or, if you have libjpeg-turbo >= 3.0:

diff --git a/setup.py b/setup.py
index f0e4f7a..15fda0c 100644
--- a/setup.py
+++ b/setup.py
@@ -594,7 +594,7 @@ def customize_build_macports(EXTENSIONS, OPTIONS):
     del EXTENSIONS['zlibng']

     # uncomment if building with libjpeg-turbo 3
-    # EXTENSIONS['jpeg8']['sources'] = []
+    EXTENSIONS['jpeg8']['sources'] = []

     EXTENSIONS['szip']['library_dirs'] = ['%PREFIX%/lib/libaec/lib']
     EXTENSIONS['szip']['include_dirs'] = ['%PREFIX%/lib/libaec/include']

[Schamschula]

The second patch worked for me!

See: https://github.com/macports/macports-ports/commit/728e4fbfe243c54733b8beeaa00940fb148bcf7b

The only hiccup was that the file doesn't use unix line endings, and macOS patch didn't tell me so. However, GNU patch did give me Hunk #1 FAILED at 594,7 (different line endings), and I ran unix2dos patch-setup.py.diff and the patch applied correctly.

As MacPorts libjpeg-turbo is at version 3.0.4, the source setup.py should have that line uncommented.

[cgohlke]

As MacPorts libjpeg-turbo is at version 3.0.4, the source setup.py should have that line uncommented.

Excellent. The extension using libjpeg-turbo has many more features.