search

rnajena / bertax_training

Training scripts for BERTax
8 stars 4 forks source link

Bertax pre-trained model #12

Closed prashexp closed 2 months ago

prashexp commented 5 months ago

Hi, I want to fine-tune the bertax pretrained model for my own dataset. Can you tell me which one is the (pre-trained) model that I can use for my task, and also is it possible to use pytorch for fine-tuning. Can you also share, if you have put out any code file for custom fine-tuning.

f-kretschmer commented 5 months ago

Hi, for our final BERtax models we used the pre-trained model bert_nc_C2_final.h5. For the whole training/fine-tuning process refer to the README (https://github.com/f-kretschmer/bertax_training#training-new-bertax-models); specifically you probably need to run the models.bert_nc_finetune script (an example call of this script is in the README) after you have brought your data into the right format.

As BERTax depends on keras-bert, with these scripts only keras/tensorflow can be used, however this architecture can of course also be used in a pytorch model, although you would have to create your own scripts for this. A starting point could be our visualize-script (https://github.com/f-kretschmer/bertax/blob/master/bertax/visualize.py), where the BERtax model is actually converted into a pytorch model in order to use visualization utilities.