search

eole-nlp / eole

Open language modeling toolkit based on PyTorch
https://eole-nlp.github.io/eole
MIT License
45 stars 9 forks source link

Pre-trained embeddings #84

Closed HURIMOZ closed 1 week ago

HURIMOZ commented 3 weeks ago

Hi guys, Iʻve gotten my hands on Eole lately. Iʻve been able to launch a couple of experiments based on your wmt17 recipe. Thanks for the recipe BTW. Iʻm however wondering whether you are planning on integrating pre-trained embeddings functionality with Eole. It would be great to have that work. We used GloVe and Fasttext pre-trained embeddings with OpenNMT back then and for low-resource languages they are of great help. I see in your documentation that you have an FAQ about GloVe embeddings but it looks like the page is from an OpenNMT template as the "train eole" command will throw a Pydantic error when trying to use pre-trained embeddings.

vince62s commented 3 weeks ago

don't expect improvement from this. FYI I imported embeddings from llama 3 and trained and encoder decoder while freezing those embeddings, results were worse.

francoishernandez commented 3 weeks ago

Yes, this doc might not be fully up to date. The feature is supposed to be working though. We retained it for the sake of facilitating potential experiments on the topic, but it's not really recommended for standard setups.

HURIMOZ commented 3 weeks ago

Weʻve used pre-trained embeddings for our bilingual models with OpenNMT (both GloVe and fasttext) and they worked like a charm, they greatly enhanced our models. I donʻt know about Llama but weʻve used OpenNMT for bilingual NMT models and LLMs have not proved better at translation for low-resource languages. Iʻm not sure what is your plan with Eole but please tell us whether you will keep developing for bilingual NMT too or not.