An external recommender cannot set a confidence score for NER prediction

[Apollo-XI]

An external recommender cannot set a confidence score. So, active learning is useless when external recommenders are used.

The method getScoreFeature in the class ExternalRecommender returns a Optional.empty(). (line 303 in ExternalRecommeder class). I think that it if a "score" feature is present should return it as OpenNLP NER Recommender does (line 115 of OpenNlpNerRecommender).

Update: External recommender can now specify their confidence by using a feature called ${feature}_score, e.g. for named entities value_score. The documentation needs to be adjusted to reflect that.

• [ ] Document how external recommender can use the score feature

[reckart]

For the internal recommenders, we use the PredictedSpan type. For external recommenders, we use the target layer type.

I guess we have kind of two options now:

• have the user add some floating point feature to the target layer and let the user choose that in the trait configuration as the score feature. That might not be a bad idea anyway - I would even go as far as adding score features in DKPro Core.
• use the PredictedSpan when communicating with external recommenders.

@jcklie @Apollo-XI any opinions?

[Apollo-XI]

First of all, I don't know much about INCEpTION code.

If PredictedSpan is something internal of INCEpTION, I wouldn't use it. However, if UIMA CAS defines a PredictedSpan type and it is easier to implement by external recommenders, I think it's a the best option. INCEpTION wouldn't treat differently an external or internal recommender. In case that using PredictedSpan makes harder to integrate an external recommender, I would avoid it.

The score feature seems fine to me if I can be set separately to each feature in the layer. Some layers could have more than one feature and the external recommender could assign different confidence scores to each one.

[jcklie]

@Apollo-XI PredictedSpan is INCEpTION internal, but I think that exposing it or using another custom Annotation just for external recommender would be the simplest solution. We consider to add a score feature to DKPro types in the future, but that is far away.

[reckart]

@jcklie I guess this can be addressed now that #1206 has been merged?

[jcklie]

Yes, we will work on that now.

[jcklie]

External recommender can now specify their confidence by using a feature called ${feature}_score, e.g. for named entities value_score.

[reckart]

@jcklie so should we mark this issue as resolved now? Does the documentation for the external recommender already reflect what you described in your comment above?

[jcklie]

The documentation does not yet reflect it, so I would leave it open.

[reckart]

Should we push it to 0.11.0 or could you update the documentation and resolve it?

[jcklie]

We right now do not have any real documentation on how to add a external recommender, only how the API looks, so I would push it.

[reckart]

We have the functionality and the documentation these days - so closing this one.

[goodcheer]

@reckart from which document can i find how to add score to external recommender? I'm now looking at implementing external recommender using python (spacy) and could'nt find one.

[reckart]

I think this should work

for cas_token, spacy_token in zip(cas.select(TOKEN_TYPE), doc):
            prediction = create_prediction(cas, layer, feature, cas_token.begin, cas_token.end, spacy_token.tag_)
            prediction.set(f"{feature}_score", YOUR_SCORE_VALUE)
            prediction.set(f"{feature}_score_explanation", SOME_STRING_WITH_ADDITIONAL_INFORMATION_OPTIONAL)
            cas.add_annotation(prediction)