-
Title.
https://huggingface.co/spaces/mteb/leaderboard
Can use llamafile for local generation: https://future.mozilla.org/builders/news_insights/llamafiles-for-embeddings-in-local-rag-applications/
-
## Detailed Description
It would be great to evaluate the live models
## Context
- good to test the different models
## Possible Implementation
- use data from [here](https://huggingface.co/d…
-
I was wondering how the trained models are intended to be evaluated. I don't believe that the paper states how many samples were used to compute the metrics. The code appears to give some indication b…
-
Not sure if this feature belongs to this library or would it require a complete separate library. I am proposing the creation of a library where llm benchmarks can be ran. For example, evaluating a mo…
-
Hi,
Appreciate the great work!
If I want to test other model performance, how to do?
e.g. To test Llama3 405 B, what data format should I pass to your interface?
Thxs!
-
Currently we are using [trocr-base-printed](https://huggingface.co/microsoft/trocr-base-printed). Identify 3 other models and evaluate our OCR pipeline using these models. The model can be changed in…
-
![image](https://github.com/sandialabs/snl-quest/assets/155320660/0a28570f-0f0b-4a53-b20a-a2870ad6358e)
![image](https://github.com/sandialabs/snl-quest/assets/155320660/931976e4-cbdf-443c-aef0-2d0c0…
-
Moving on to the **benchmark_models** tool in `evaluate_llm_tool.py,` and I had a few suggestions that might help improve it:
- Docstring Accuracy:
The docstring currently mentions a prompt_set…
-
How can we integrate the use of language models to evaluate language model generations?
Currently, lm-eval evaluates language model generations with conventional metrics such as accuracy, bleu, etc…
-
[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
** Facing error with using Langchain wrapped hugging face models**
I am …