search

MantisAI / nervaluate

Full named-entity (i.e., not tag/token) evaluation metrics based on SemEval’13
MIT License
154 stars 19 forks source link

Evaluator accept prodigy model instead of y_true, y_pred #47

Closed nsorros closed 3 years ago

nsorros commented 3 years ago

As Prodigy does not offer a quick command to do precautions, it would be convenient to just be able to pass the data and model to Evaluator and get the metrics instead of having to run a predict script to use the tool.

nsorros commented 3 years ago

@ivyleavedtoadflax have I missed something here?

ivyleavedtoadflax commented 3 years ago

No, you're right, but in the short term I would prioritise fixing the bugs over implementing a model API

nsorros commented 3 years ago

I want to do these improvements so that we can have cube using it for evaluation

ivyleavedtoadflax commented 3 years ago

agree :+1

nsorros commented 3 years ago

I think this will be covered by #44 implemented as described in #49 so closing for now