trie split bug

[zpp13]

System Info

• transformers version: 4.41.0
• Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.23.4
• Safetensors version: 0.4.3
• Accelerate version: 0.30.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.1+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker and @itazap

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

from transformers.tokenization_utils import Trie trie = Trie() trie.add("abc") trie.add("b") trie.split("ab cd")

['ab c', 'd']

Expected behavior

in my opinion, this should get ['a', 'b', 'cd'],but get ['ab c', 'd'] First submission, sorry if I understand something wrong.

[ArthurZucker]

Hey! The trie is mostly for internal usage, I think this is expected ! (we don't take spaces into account AFAIK)

[zpp13]

This error occurs not only with spaces,for example:

[zpp13]

@ArthurZucker and @itazap

[ArthurZucker]

Hey, sorry but the tre is not for general purpose usage, if it is breaking a tokenizer than sure let's fix it, but it does not seem to be the case!