search

asahi417 / tner

Language model fine-tuning on NER with an easy interface and cross-domain evaluation. "T-NER: An All-Round Python Library for Transformer-based Named Entity Recognition, EACL 2021"
https://aclanthology.org/2021.eacl-demos.7/
MIT License
373 stars 40 forks source link

The outcomes of the two running methods are different. #55

Open wangChangHai1 opened 5 months ago

wangChangHai1 commented 5 months ago

[Uploading question.docx…]()

wangChangHai1 commented 5 months ago

Hello, could you explain why the outcomes of these two running methods differ? What kind of segmentation tool was used in the first method?

https://huggingface.co/tner/deberta-v3-large-btc Inference API

from tner import TransformersNER model = TransformersNER("tner/deberta-v3-large-btc") model.predict(["Congratulations Local 330 members who were promoted this morning! Steve Kaut to Deputy Chief, Todd Kelly to District Chief, Matt Askea and Brent Scarbrough to Captain, and Will Carter, Jeff Gostkowski, and Dave Goodspeed to Lieutenant. Good luck in your positions. #330forthe330 https://t.co/acHXx6DRY4"])

asahi417 commented 5 months ago

First of all, the model is deberta large model fine-tuned on BTC dataset https://huggingface.co/datasets/tner/btc, and we fine-tune models on NER task with CRF layer https://towardsdatascience.com/conditional-random-fields-explained-e5b8256da776, which is not available on the inference API at the moment, so when we call the model via inference API, it runs the model with out the CRF layer, which results in the different output from the tner API. In short, CRF layer adds bias of the sequence dependencies (eg. I-PER should come after B-PER) to fix the prediction, so you can expect the prediction from inference API is slightly worse than what you can get from tner API due to the CRF availability.