NER Inference always returns empty list

[felixvor]

Question Hello,

I trained an NER model on my own dataset (about 30k labeled sentences) and the evaluation looks very promising (Test set has about 93% accuracy on 18 different entity types). However, whenever I run inference, i always get an empty predictions list. I tried many examples from my train and test set and never got any prediction.

This is not the case when I use the model that was trained with your ner example. Here inference works fine. I used the debugger to compare the processed data of your example with mine and it looks the same (key is "text" with the correct "ner_label" list).

I was able to replicate the problem by creating a simplified dataset with 25 examples and a single entity type. I overfit the model on this dataset to 100% accuracy and try to run inference on an example from the training data. Here is how the data format looks like:

-DOCSTART-  O
Ich O
bin O
7   number
.   O

-DOCSTART-  O
Mein    O
Alter   O
ist O
12  number
.   O
. . .

When I run:

    basic_texts = [
        {"text": "Mein Alter ist 12 ."},
    ]
    model = Inferencer.load(save_dir)
    result = model.inference_from_dicts(dicts=basic_texts)
   print(result)

I get:

   [{'task': 'ner', 'task_name': 'ner', 'predictions': [[]]}]

The Code I use for training is here. The only thing that is different from your working example is the dataset, the label_list, the delimiter and num_epochs. I also tried changing the delimiter and renaming the labels to fit to your example but it did not help.

Here is a full dump of the training process that shows correctly looking training samples, 100% accuracy on test set, followed by empty inference prediction on an example from the training set.

Here is the simplified dataset I used to reproduce the problem.

I hope I am not overlooking something obvious here. I you have any idea for what else I could look into to fix this, it would be very much appreciated!

[FabriceEll]

I came across this issue as well a couple months ago when using FARM in my Windows work environment, ended up not using NER because of this

[julian-risch]

Hi @DieseKartoffel and @FabriceEll I can have a look at this problem but it will take some time(~2 weeks). Maybe this test case helps if you look into it yourselves. The test case uses another model, which also works: https://github.com/deepset-ai/FARM/blob/81d1b31791a213be17709cf9d46f9c9981579e34/test/test_conversion.py#L156

[julian-risch]

Could you please try loading the inferencer with inferencer = Inferencer.load(save_dir, task_type="ner", num_processes=0)? If the inferencer is the problem, maybe you can find out whether the storing of the model and loading of the inferencer does work correctly?

[felixvor]

Hello Julian, thank you for looking into this! Hopefully we can find the problem soon. I didn't know about the conversion test so I used the code from there. I also initialized the inferencer with task type and without multiprocessing (tried linux and windows) but never got any predictions. However I found that the prediction head returns the correct probabilites for my inference example! At the corresponding word positions all_probs contain high values at the correct label index. So they must be lost somewhere along the way from there. https://github.com/deepset-ai/FARM/blob/81d1b31791a213be17709cf9d46f9c9981579e34/farm/modeling/prediction_head.py#L672

EDIT: I looked further into it and finally found the place where the predictions are lost! The function formatted_preds of the ner prediction head has alle the correct data (correct word spans, correct labels from label_list and their probabilites) in preds_seq, spans_seq, probs_seq. The data gets passed to convert_iob_to_simple_tags which returns the empty lists. The function contains string comparisons to "B-" and "I-" which I do not use in my dataset.

https://github.com/deepset-ai/FARM/blob/81d1b31791a213be17709cf9d46f9c9981579e34/farm/modeling/prediction_head.py#L708

It seems as Farm is currently not compatible with datasets that use their own entity naming convention. I want to try and fix this and get a PR ready if possible. Can you give me some context about what convert_iob_to_simple_tags is supposed to do exactly?

[julian-risch]

@DieseKartoffel Great insights! You identified the problem. IOB stands for Inside–Outside–Beginning and is a tagging format. FARM uses the IOB2 Format: https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging) B marks the first token (beginning) of an entity, I marks a token within an entity and O is a token outside of an entity. Example:

Alex B-PER is O going O to O Los B-LOC Angeles I-LOC in O California B-LOC

convert_iob_to_simple_tags converts the IOB2 format to a format without IOB markers (plain text with spans). By the way, FARM also has a function that can convert IOB to IOB2 format: https://github.com/deepset-ai/FARM/blob/81d1b31791a213be17709cf9d46f9c9981579e34/farm/data_handler/utils.py#L275 Very much looking forward to your PR!

[julian-risch]

closed by #828