Research Priority Queue

[iejMac]

Upwards Distillation (L -> H):

• [ ] MSE/Loss between similarities vs. embeddings (https://github.com/iejMac/encoder-distill/issues/14)
• [ ] Test out contrastive fine tuning without removing MLP, if improves, try to bring down teacher dim instead of student up (Bring teacher down teacher dim. instead of student up (https://github.com/iejMac/encoder-distill/issues/15)
• [ ] Try to improve loss between similarities by applying distillation lower in the student network, allowing more room for improvement (https://github.com/iejMac/encoder-distill/issues/16)

Downwards Distillation (L -> B):

• [ ] Bring teacher down teacher dim. instead of student up (https://github.com/iejMac/encoder-distill/issues/15)