-
I use torch.profiler.profile() to profile mixtral based on vllm. And I found lots of blank before each runing step.
![S85Z22{PW)GZ0(E)4AH4AF1](https://uploads.linear.app/342cff15-f40f-4cf7-8bee-343d2…
-
### What happened?
llama.cpp使用QWen2.5-7b-f16.gg在310P3乱码
### Name and Version
./build/bin/llama-cli -m Qwen2.5-7b-f16.gguf -p "who are you" -ngl 32 -fa
### What operating system are you seeing the …
-
**Describe the bug**
When a provider is explicitly set on `defaultTest` or through the command using `--grader`, some random assertions are failing with the error message "Could not extract JSON from…
-
### Your current environment
vllm version: 0.6.3.post1
### Model Input Dumps
_No response_
### 🐛 Describe the bug
I see on the official site of gemma: https://huggingface.co/google/gemma-2b, cont…
-
[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
**Your Question**
faithfulness_score: always be nan
**Code Examples**…
-
Hi,
Thank you for sharing your impressive work! Equipping LLMs with temporal understanding is indeed a challenging task. I have a question regarding the ActivityNet results:
Are the scores you r…
-
Thank you very much for doing such great open-source work!
i try:
CUDA_VISIBLE_DEVICES=X bash scripts/evaluate.sh PATH_OR_NAME_TO_BASE_MODEL PATH_TO_SAVE_TUNE_MODEL PATH_TO_PRUNE_MODEL EPOCHS_YOU…
-
It would be neat to give the LLM the ability to interact with the current web page. User should be able to describe some page interaction and the LLM executes it. This is likely some combination of …
-
-
@haileyschoelkopf @lintangsutawika @baberabb
The following is a list of TODOs to implement LLM-as-a-Judge in Eval-Harness:
**TLDR**
* Splits existing `evaluate` function into `classification_e…