Build BertHead

[pmhalvor]

Checklist:

• [x] init BertModel object from pretrained
• [x] output layers for each task
• [x] optimizers for each task
• [x] predictions for each task
• [x] evaluation logs for each task
• [x] run long model over night

[pmhalvor]

Some things to note about this model:

• Holder seems to be learning well, across batches
• Target and polarity end up syncing up around 20 epochs for both 75% and 50% of data, w/ some batch-wise fluctuations
  • Should probably have individual learning rate
• Expression shows very much batch-wise fluctuation, with little learning in the beginning
  • Could mean learning rate too high already from start
• Learning rate schedulers are made, but never turned on
  • First task today should be to learn how to activate these

[pmhalvor]

Another point to note is that no Linear layer has dropout here. To regularize a bit, a dropout can be provided in init_components()

[pmhalvor]

Another point to note is that no Linear layer has dropout here. To regularize a bit, a dropout can be provided in init_components()

Linear layers don't have dropouts.