Code not producing expected entities using spacy_llm

[AndreaLombax]

Hi, I was trying out the new NERv2 task from spacy-llm. I built the following code by concatenating the various examples in the different spacy sources.

from spacy_llm.util import assemble
nlp = assemble(
        "config.cfg",
        overrides={}
    )
doc = nlp("Jack and Jill went up the hill")

print(f"Text: {doc.text}")
print(doc.ents)
print(f"Entities: {[(ent.text, ent.label_) for ent in doc.ents]}")

The code should recognize named entities in the input text and display them in the output. But the output is the following:

> Text: Jack and Jill went up the hill
> ()
> Entities: []

This is the config file:

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = ["PERSON", "ORGANIZATION", "LOCATION"]

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-3b"

[components.llm.task.label_definitions]
PERSON = "Extract any named individual in the text. This entity refers to a specific person who can be identified by name or title."
ORGANIZATION = "Extract any named individual in the text. This entity refers to a specific person who can be identified by name or title."
LOCATION = "Extract any named geographical location or place in the text. This entity represents the names of cities, countries, landmarks, or any other identifiable geographic areas."

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "ner_examples.json"

Also, I tried the same code using this pattern:

[components.llm.model]
@llm_models = "spacy.Dolly.v1"
name = "dolly-v2-7b"

But I got the same output (after 7.5 seconds).

Anyone has the same issue?

[AndreaLombax]

UPDATE:

I tried several modifications, but no results. To ward off any memory problems, I removed the few shot examples and continued testing with different texts. The code remained unchanged.

Until I tried with this text and got this (empty) output:

> Text: In New York City, Dr. Emily Parker, a renowned neuroscientist, worked at the NeuroTech Institute. She presented her groundbreaking research at the Global Science Alliance conference in London. Her work attracted collaboration offers from Professor Zhang Wei in Beijing. Dr. Parker's influence extended worldwide, with invitations to speak in Sydney and Cape Town, shaping the future of neuroscience.
> ()
> Entities: []

I don't know why, but I tried restarting the exact same code without any changes, and this happened:

> Text: In New York City, Dr. Emily Parker, a renowned neuroscientist, worked at the NeuroTech Institute. She presented her groundbreaking research at the Global Science Alliance conference in London. Her work attracted collaboration offers from Professor Zhang Wei in Beijing. Dr. Parker's influence extended worldwide, with invitations to speak in Sydney and Cape Town, shaping the future of neuroscience.
> (New York City, Dr. Emily Parker, NeuroTech Institute, Global Science Alliance, Professor Zhang Wei)
> Entities: [('New York City', 'LOCATION'), ('Dr. Emily Parker', 'PERSON'), ('NeuroTech Institute', 'ORGANIZATION'), ('Global Science Alliance', 'ORGANIZATION'), ('Professor Zhang Wei', 'ORGANIZATION')]

And now I've noticed that some times I have to submit the text multiple times to get an output, it seems almost random.

[rmitsch]

Hi @AndreaLombax! Most of our prompts are tested against GPT-3.5 or a similarly capable model. My assumption is that Dolly (whether 3b or 7b) doesn't return a parsable output, i. e. on that's in line with what the prompt specifies. Some models require more precise prompting than others, especially when it comes to the output format.

I recommend logging the model outputs and posting the model's output here.

[rmitsch]

I don't know why, but I tried restarting the exact same code without any changes, and this happened:

LLMs are not deterministic. If you're looking for determinism, you can lower the temperature or some equivalent parameter to 0. In this case the LLM likely sometimes produces parsable and sometimes non-parsable output .

[AndreaLombax]

I don't know why, but I tried restarting the exact same code without any changes, and this happened:

LLMs are not deterministic. If you're looking for determinism, you can lower the temperature or some equivalent parameter to 0. In this case the LLM likely sometimes produces parsable and sometimes non-parsable output .

You are right, but changing the temperature brought me to the same results. I'll wait to get a larger GPU and retry using the same code. Thank you very much!