Currently, beir evaluates a model by calling its encode_queries and encoder_corpus functions. These in turn call the forward methods of the model.
This works great for pytorch-based models, but not for models from other deep-learning frameworks.
To circumvent this issue, I've introduced a "general" offline model which is a static numpy array of query and corpus representations along with a mapping of desired corpus-ids for each query-id.
Thus, any model from any framework can dump its representations for queries and corpus in a numpy format which can then be evaluated using this PR.
Currently, beir evaluates a model by calling its encode_queries and encoder_corpus functions. These in turn call the
forwardmethods of the model.This works great for pytorch-based models, but not for models from other deep-learning frameworks.
To circumvent this issue, I've introduced a "general" offline model which is a static numpy array of query and corpus representations along with a mapping of desired corpus-ids for each query-id.
Thus, any model from any framework can dump its representations for queries and corpus in a numpy format which can then be evaluated using this PR.