-
Hello 👋
First of all thank you for the great work and evaluation results!
I have understood that in many cases you predicted outputs for each question based on the choice that minimizes the loss…
-
Dear authors:
It's very promising to witness that the stronger Mistral-7b llm models enhance the capability of video understanding. We would eager to see more potentials performed by replacing the …
-
[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
** How do I specify the name of the model**
An LLM is deployed locally. …
-
[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.
**Describe the bug**
Getting "asyncio.exceptions.CancelledError" as an error
Ra…
-
# Proposed Feature
Add an efficient interface for generation probabilities on fixed prompt and completion pairs. For example:
```python
# ... load LLM or engine
prompt_completion_pairs = [
…
-
## Evaluating (open) LLMs for more languages using translated prompts
As part of MPEP, we are now at the point where some translation efforts have successfully translated 500 highly ranked prompts …
-
I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.
**Describe the bug**
I am trying to run the template code from the Github ReadMe page.…
-
The Issue discusses the Creation of the Evaluation Script.
The following needes to done as part of the Evaluation Script.
- Use the language prompts that are present in the [prompt](https://github…
-
### System Info
- nvidia:535.129.03
- cuda_version:12.4
- GPU:L40S
- OS:Ubuntu 22.04.4 LTS(docker)
- tensorrt-llm: 0.11.0.dev2024060400
### Who can help?
_No response_
### Information
…
-
https://github.com/SafeAILab/EAGLE/blob/35854779f60db028ec646d945cacbdebdce490bb/eagle/evaluation/gen_ea_alpha_vicuna.py#L93
It looks like a bug? The calculation of accept_length only considers the…