Crashes with `tokenizers.cpython-311-x86_64-linux-gnu.so: undefined symbol: OnigDefaultSyntax`.

[dreirund]

Ahoj,

since I am not knowledgeable a lot into python packages and even less into linking and dynamic libraries stuff, I don't know how to research this more or where to report it properly. So I report it here, maybe someone/ the maintainer knows more and can redirect.

When I launch sdiff-gtk (on Artix GNU/Linux) or python /usr/lib/python3.11/site-packages/sdiff_gtk/__init__.py, it fails with
/usr/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-x86_64-linux-gnu.so: undefined symbol: OnigDefaultSyntax:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1382, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.11/site-packages/transformers/models/__init__.py", line 15, in <module>
    from . import (
  File "/usr/lib/python3.11/site-packages/transformers/models/mt5/__init__.py", line 36, in <module>
    from ..t5.tokenization_t5_fast import T5TokenizerFast
  File "/usr/lib/python3.11/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 24, in <module>
    from ...tokenization_utils_fast import PreTrainedTokenizerFast
  File "/usr/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 25, in <module>
    import tokenizers.pre_tokenizers as pre_tokenizers_fast
  File "/usr/lib/python3.11/site-packages/tokenizers/__init__.py", line 78, in <module>
    from .tokenizers import (
ImportError: /usr/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-x86_64-linux-gnu.so: undefined symbol: OnigDefaultSyntax

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/diffusers/utils/import_utils.py", line 710, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 20, in <module>
    from transformers import CLIPImageProcessor, CLIPTextModel, CLIPTokenizer, CLIPVisionModelWithProjection
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/usr/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1372, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1384, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.clip because of the following error (look up to see its traceback):
/usr/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-x86_64-linux-gnu.so: undefined symbol: OnigDefaultSyntax

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/sdiff_gtk/__init__.py", line 6, in <module>
    from diffusers import StableDiffusionPipeline
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/usr/lib/python3.11/site-packages/diffusers/utils/import_utils.py", line 701, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/diffusers/utils/import_utils.py", line 701, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/diffusers/utils/import_utils.py", line 700, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/diffusers/utils/import_utils.py", line 712, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion because of the following error (look up to see its traceback):
Failed to import transformers.models.clip because of the following error (look up to see its traceback):
/usr/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-x86_64-linux-gnu.so: undefined symbol: OnigDefaultSyntax

I have installed the latest state from this git repository (git commit hash: 689cba6), using the sdiff-gtk-git AUR package.

Regards!

[dreirund]

Also happens with sdiff-gtk release 0.1.0 and 0.1.1.

[dreirund]

Closing as ↗ the fix is to build python-tokenizers without LTO.