Multiple INFIX inside span cannot be recognized

[nsch0e]

A span containing more than one INFIX "tokens" will not be recognized.

Expected behaviour

all 4 date strings should be recognized.

How to reproduce the behaviour

Was run in a jupyter notebook for displaying purpose

from spacy.tokenizer import Tokenizer
import re
import spacy

records = [
    ("10/02/2015", {"spans": {"sc": [(0, 10, "DATE")]}}),
    ("10/02.2015", {"spans": {"sc": [(0, 10, "DATE")]}}),
    ("10/2015", {"spans": {"sc": [(0, 7, "DATE")]}}),
    ("10.2015", {"spans": {"sc": [(0, 7, "DATE")]}}),
]
infix_re = re.compile(r"""[/\.]""")

def custom_tokenizer(nlp):
    return Tokenizer(nlp.vocab, infix_finditer=infix_re.finditer)

model = spacy.blank("EN")
pipe = model.add_pipe("spancat")
pipe.add_label("DATE")

model.tokenizer = custom_tokenizer(model)
model.initialize()

trainData = [spacy.training.Example.from_dict(model.make_doc(t), a) for t, a in records]
optimizer = model.begin_training()
for i in range(1000):
    model.update(trainData, sgd=optimizer, drop=0.2)

for t, a in records:
    spacy.displacy.render(model(t), style="span", jupyter=True)
    print(model.tokenizer.explain(t))

Your Environment

• spaCy version: 3.7.4
• Platform: Linux-6.5.0-28-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• Pipelines: en_core_web_lg (3.7.1), en_core_web_md (3.7.1)

[nsch0e]

Sry, was a user error: the default suggester for spans uses ngrams of sizes 1,2,3. The bigger spans contain 5 tokens, so the suggester never suggested the complete span.

Fixed by using this config:

config = {
    "suggester": {"@misc": "spacy.ngram_suggester.v1", "sizes": [1, 2, 3, 4, 5]}
}
pipe = model.add_pipe("spancat", config=config)

[svlandeg]

No problem, thanks for reporting back! 🙏

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.