Can sentence-transformers easily adapt to listwise loss function (e.g. lambdarank)?

[shangh1]

Hi, I'd like to run a cross-encoder model training with a listwise loss function. Do you have any suggestions whether there is an easy way for this?

The input data is (query, item) level. To train with listwise loss, I need to

• (1) have all items corresponding to the same query belong to the same batch. If a query q have 5 corresponding items i1, i2, i3,i4,i5, then we need (q,i1),(q,i2),(q,i3),(q,i4),(q,i5) are in the same batch.
• (2) for the loss function in this line (https://github.com/UKPLab/sentence-transformers/blob/737353354fbdf1a419eee864f998ffe9fdf3b682/sentence_transformers/cross_encoder/CrossEncoder.py#L240), we'll need another argument to denote the query, so items corresponding to the same query can be used for computing listwise loss.

Thanks, Hongwei

[tomaarsen]

Hello!

I think it might make more sense to implement this in a DataLoader or collator which are responsible for loading the data. That way you always have confidence that the values in the loss function correspond with the correct batch.

• Tom Aarsen