search

flairNLP / flair

A very simple framework for state-of-the-art Natural Language Processing (NLP)
https://flairnlp.github.io/flair/
Other
13.85k stars 2.09k forks source link

[Question]: Gain to set use_context = True for fine tuning if the data is not at a document level. #3551

Closed abhirupr closed 2 hours ago

abhirupr commented 3 hours ago

Question

Hello,

I have a custom annoted data for a set of addresses with tags of STREET, CITY and COUNTRY, I want to fine tune transformer embeddings as proposed in FLERT so that it can predict these tags from any address (as a pre trained FLAIR model can only provide the tag LOC).

Given the data is not at a document level (i.e it is a set of addresses and the preceeding and subsequent address of an address has no relation with it), if I have to fine tune embeddings of xlm-roberta-large (as used in the paper), do I have to set use_context = True? What can be the advantages of setting it True in this case?

Also, if context is not required, instead of using a transformer based embeddings can I use something more simpler like GLoVe?

alanakbik commented 3 hours ago

Hi, if there is no document-level context, you can set use_context=False. This will likely produce better results and also run faster.

Transformer-based embeddings will be significantly better than GloVe. But if you want to optimize speed, you can either look at using a smaller transformer, or using the classic FlairEmbeddings setup.

abhirupr commented 2 hours ago

Thank you. I believe transformer based embeddings take into account the position which will be helpful in this case as usually the STREET is followed by CITY and the COUNTRY.