-
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Topic
A lot of user is curious about how fast those embeddings are and how it can be improved.
We should ad…
-
Evaluation failed: 'CustomOllama' object has no attribute 'set_run_config', what is the solution,
Ragas Version: 0.1.7
**Code Examples**
# Define a simple dataset using Pandas DataFrame
data…
-
### 🚀 The feature, motivation and pitch
With a single command, quantize the same model across every available quant scheme and configuration and output a table that compares the results. This will …
-
# Input Size Mismatch in `goat_skill_chain` Evaluation
## Problem Description
When running the evaluation of `goat_skill_chain`, two main issues are encountered:
1. The checkpoint file `ckpt.…
-
Running the tasks with `BAAI/bge-visualized-base-base/m3` and getting errors like below
```
ERROR:mteb.evaluation.MTEB:Error while evaluating InfoSeekIT2TRetrieval: The size of tensor a (516) must m…
-
Does the evaluation support multi GPU? I used the script to evaluate bge-m3 on one task, and it takes more than 10 hours on one GPU. p.s. I tried to enlarge the batch_size, but always leads to OOM.
`…
-
i deifned this function to not use the LLM calls but it still give me this error:
Exception has occurred: AttributeError
'list' object has no attribute 'get_sample_type'
File "C:\Users\Amin\One…
-
KeyError: 'C0002962'.
-
## Issue encountered
It would be good to have a system for evaluating both the relevance of the RAG and its use by the LLM in producing the response. My first intuition would be a multi-stage system …
-
### Objectives
- Develop and evaluate alternative plasmid embedding methods to serve as baselines for the OpenPlasmid dataset
- collaborate on establishing metrics
- related to #1
### Tasks
- …