search

emanjavacas / pie

A fully-fledge PyTorch package for Morphological Analysis, tailored to morphologically rich and historical languages.
MIT License
22 stars 10 forks source link

Try to add context embeddings #38

Open PonteIneptique opened 4 years ago

PonteIneptique commented 4 years ago

Hey Enrique, I want to dig in Contextual embeddings/Language models and in particular Bert. I want to see if the these features would help the lemmatizer for languages close to French (Camembert).

Did you plan to do it ? If so, please tell me, I would not want to steal that from you :)

emanjavacas commented 4 years ago

Hey! This went missing... sorry! In the current setup, if you use the LM loss, you are effectively leveraging contextual embeddings (even before they were cool). Of course you are missing the effect of pre-training on massive datasets. Feel free to explore it. I could also help if you want. For lemmatization, however, I wont expect miracles, since the possibilities in which contextual information helps there are limited (mostly disambiguation).

PonteIneptique commented 4 years ago

Well, I actually did it :) I am figuring some things out, and discovered few bugs with CRF (probably breaking changes that went in at some point...)