BertMesh meets transformers

[nsorros]

Description

Even though we have upload BertMesh model into the hugging face hub here the weights are not initialised properly and additionally making it work required a couple of manual tweaks.

A more standard way to get a custom model into the hub is to subclass PreTrainedModel instead of torch.nn.Module which gives you access to save_pretrained and load_pretrained. It seems that PreTrainedModel is itself a wrapper on top nn.Module with some extras so it should not restrict us too much hopefully 🤞

This is ready to run @aCampello. Note you need to recreate the environment

make virtualenv
dvc repro grants_tagger/bertmesh/dvc.yaml

Also you need to get weights and biases creds from Nick. This needs to run in g4dn.metal 🔥

Checklist

• [ ] Linked to Notion or GitHub issue
• [x] Added tests
• [ ] Updated README
• [ ] DVC up to date

[aCampello]

Happy to run this, just let me know how to set up the credentials

[nsorros]

One thing that needs to happen is to update to the latest WellcomeML after https://github.com/wellcometrust/WellcomeML/pull/384 is merged which will allow us to get the latest version of transformers which enables one functionality needed to make it easier to upload to the hub.

[aCampello]

It seems to me that the model on the model hub is still the old one:

https://huggingface.co/Wellcome/WellcomeBertMesh/tree/main

[aCampello]

The hub is also a bit weird, it says to use it we need to:

from transformers import AutoTokenizer, BertMesh

tokenizer = AutoTokenizer.from_pretrained("Wellcome/WellcomeBertMesh")

model = BertMesh.from_pretrained("Wellcome/WellcomeBertMesh")

Should it be BertModel instead?

[nsorros]

The hub is also a bit weird, it says to use it we need to:

from transformers import AutoTokenizer, BertMesh

tokenizer = AutoTokenizer.from_pretrained("Wellcome/WellcomeBertMesh")

model = BertMesh.from_pretrained("Wellcome/WellcomeBertMesh")

Should it be BertModel instead?

You can do BertMesh but then you need the code. Instead you can now do

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Wellcome/WellcomeBertMesh")
model = AutoModel.from_pretrained("Wellcome/WellcomeBertMesh", trust_remote_code=True)

It seems to me that the model on the model hub is still the old one:

https://huggingface.co/Wellcome/WellcomeBertMesh/tree/main

Why do you say that?

[nsorros]

Labels added in config so now you can get labels straight out of the model. Try it

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Wellcome/WellcomeBertMesh")
model = AutoModel.from_pretrained("Wellcome/WellcomeBertMesh", trust_remote_code=True)

inputs = tokenizer(["This grant is about malaria and not about HIV."], padding="max_length")
labels = model(**inputs, return_labels=True)
print(labels)

[aCampello]

I can confirm that this works for me now

[nsorros]

I can confirm that this works for me now

Did you approve as well? :D

[aCampello]

I can confirm that this works for me now

Did you approve as well? :D

Seems that black has a problem at the moment?