Open Ch-rode opened 2 years ago
Hi! Do you specifically want to re-implement bert-base, or just a transformer? I have code to train a version of ESM-1b here. This code scales better and will also result in better performance.
In that repo, the data processing is done in these lines. The masking code is then implemented in this class.
I have a bunch of utilities implemented in github.com/rmrao/evo, if it's helpful.
If you specifically want the masking code from TAPE, it's implemented here.
Hope this helps!
Hello ! Thanks for your informations. I would like to re-implement bert-base for Sequence Classification task.
Hello ! I'm trying to implement bert-base but I have not clear how do you generate the masks with the TapeTokenizer. This is my code
But my output (for example) will have only token ids (no attention mask and no possibility to set max_length or padding). How does it works? Thanks