gbert - parameter tuning - "round2 train set"

[dominikmn]

Ideas to tune gbert on the round2 train set.

Targets:

• Surpress overfitting
• Tune overall scores

Is there the chance to make a 'coarse to fine search' on those points below which are hyperparameters?

Promising

• additional linear layer / relu layer
• different learning rates
• early stopping function

Maybe

• different drop-out rate
• different random seeds (= different weight initializations and/or different data order)
• different pos_weight
• emoji tokenization

Less promising

• different optimizer
• gbert-large