Cross-encoder distillation

lightonai / pylate

Late Interaction Models Training & Retrieval

https://lightonai.github.io/pylate/

MIT License

158 stars 7 forks source link

Closed raphaelsty closed 2 months ago

raphaelsty commented 4 months ago

Working on cross-encoder distillation

We want to distill cross-encoder into ColBERT,

Update the training so we can distill logits pre-computed on option.
Create a function which computes logits using a pre-trained cross-encoder in order to create a dataset with logits that can be used to distill soft labels to colbert.

NohTow commented 2 months ago

I think we can close this now.