michaelfeil
commented
2 weeks ago Correct, that is currently not possible, but easy to implement. You are welcome to contribute this.
Task:
- Add a similar integration as in: https://github.com/michaelfeil/infinity/blob/65afe2b3d68fda10429bf7f215fe645be20788e4/libs/infinity_emb/infinity_emb/transformer/embedder/sentence_transformer.py#L87C9-L90C14
- add a test (verifying that the quantization of a reranker model works). Please use BGE-reranker BASE or mixedbread-xsmall reranker for the test
Feature request
--dtype support for rerankers
Motivation
Easily quantize cross encoder models
Your contribution
. (will look into this)