HuggingFace Transformers for SciBERT Fine Tuning

[ivalexander13]

After looking through example code for relation extraction tasks, writing something to do this task is too monumental. Now we're trying to do NER with HuggingFace, with the advantage (over AllenNLP) of having full control and transparency of the code.

Current Actions: NER on Transformers

Looking at example NER code to find a good base skeleton code. Then study AllenNLP's NER design to see any specific fine-tuning steps that we might need to replicate. Finally, begin writing code that fits our needs.

[ivalexander13]

Code for NER has been implemented with default Trainer class. Rough hyperparameter tuning has been done and results are logged in this Google Sheet. The hyperparameters tuned are:

• learning rate: 2e-5
• batch size: 1
• epoch #: tbd (probably 11?)

To-do's - expand on each:

• [ ] Save metrics to automate performance evaluation.
• [ ] Create a SLURM-friendly script. Do we need to use parallelization?
• [ ] Why is overfitting so quick?? Epoch 2 and validation loss already skyrocketed. See: Regularization. (ref)
• [ ] Include offset mapping to account for the one-to-many encoding of words.
• [ ] Figure out a way to better tune the training loop (how to dig into huggingface's model() function? How to set weights and loss functions?)
• [ ] Huggingface for RE: find a good example that isn't too convoluted.

[ivalexander13]

[To-do] Regularizations:

• Early stopping (huggingface docs) (reading)
  • use trainer's callbacks
• dropouts (reading)

Links:

• Subclassing Trainer to override methods (hf-docs)
• More discussion about NN design (huggingface, TF, keras) (tds)