jonathanbratt / RBERT

Implementation of BERT in R
Apache License 2.0
158 stars 19 forks source link

https://github.com/bnosac/golgotha #52

Open jwijffels opened 4 years ago

jwijffels commented 4 years ago

Just a note. I've created this: https://github.com/bnosac/golgotha in order to easily use the BERT embeddings in some downstream predictive models and when I tried RBERT I couldn't get the multilingual model to work. It was also a trial to see on speed of getting these embeddings and to see what these model outputs provide and how I could maybe develop this directly using libtorch bypassing python.

jonathanbratt commented 4 years ago

Thanks for the note! I don't think we've tested the multilingual models specifically in RBERT. What's the nature of the failure you encountered? Were you able to get any of the English models to work (at least formally, even if it's not suited for the task)?

jwijffels commented 4 years ago

yes, english worked fine, can't remember the exact error on the multilingual model, i believe it was already at download_BERT_checkpoint

jwijffels commented 4 years ago

is there any interest from one of you 2 to work directly from libtorch instead of using python?

jonathanbratt commented 4 years ago

Possibly! We're working right now on using tensorflow hub (via tfhub package) to load the models, but are open to additional/other ways of proceeding. I haven't had much time lately to devote to this, but hope to soon.

jonthegeek commented 3 years ago

Update: Almost definitely! We're finally actively working on this again, and the path will definitely be along the lines of "get the TF2 version working, then investigate {torch}."

jwijffels commented 3 years ago

+1 for some torch developments regarding NLP. I was thinking about some named entity recognition models and started experimenting a bit myself at https://github.com/DIGI-VUB/torch.ner but certainly having BERT embeddings through torch would be a big plus

jonthegeek commented 3 years ago

It's likely that we'll make a separate repo for torch development (although that isn't 100% decided yet). I'll definitely ping you when we figure it out!