XLMRobertaTokenizer attribute has disappeared from transformers.models.xlm_roberta

[jmalbarranORA]

System Info

Oracle Linux 9.4 (Oracle-Linux-9.4-2024.07.31-0) Python 3.12.5 (using PyEnv and .venv environment) transformers==4.44.2

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

Install Oracle OML4Py 2.0 client library as described here Install tranformer library with pip install transformers

After install, start python and execute import oml.utils We get

Python 3.12.5 (main, Aug 21 2024, 09:28:16) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3.0.1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import oml.utils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/opc/vector/.venv/lib/python3.12/site-packages/oml/__init__.py", line 77, in <module>
    from oml.utils import *
  File "/home/opc/vector/.venv/lib/python3.12/site-packages/oml/utils/__init__.py", line 23, in <module>
    from .embeddings import EmbeddingModelConfig,EmbeddingModel
  File "oml/utils/embeddings.py", line 28, in init oml.utils.embeddings
  File "/home/opc/vector/.venv/lib/python3.12/site-packages/oml/utils/_pipeline/__init__.py", line 22, in <module>
    from .PipelineBuilder import PipelineBuilder
  File "oml/utils/_pipeline/PipelineBuilder.py", line 25, in init oml.utils._pipeline.PipelineBuilder
  File "oml/utils/_pipeline/steps.py", line 38, in init oml.utils._pipeline.steps
  File "/home/opc/vector/.venv/lib/python3.12/site-packages/oml/utils/_onnx_export/__init__.py", line 22, in <module>
    from .tokenizer_export import export_tokenizer
  File "oml/utils/_onnx_export/tokenizer_export.py", line 82, in init oml.utils._onnx_export.tokenizer_export
  File "/home/opc/vector/.venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 1596, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module transformers.models.xlm_roberta has no attribute XLMRobertaTokenizer

Expected behavior

Import sucessfully, with no error

[ArthurZucker]

Hey! i think this should be opened on the repo of oml! 🤗

[jmalbarranORA]

Sure! Already did, and they indicate the transformers version to use. But, why transformers.models.xlm_roberta has removed the XLMRobertaTokenizer attribute, creating an incompatibility?

[ArthurZucker]

Modeling file never have tokenizers linked to them, so not really sure what is happening here!

[jmalbarranORA]

Neither me, but, for example, the attribute XLMRobertaTokenizerFast still exists (only XLMRobertaTokenizer has disappeared)

[ArthurZucker]

Ah, are you sure you have sentencepiece installed?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.