-
FYI I'm seeing a lot of these "errors" in my logs when running lighteval
```
Could not initialize the JudgeOpenAI model:
[Errno 2] No such file or directory: 'src/lighteval/tasks/extended/mt_benc…
-
"To evaluate the ability of LLMs to identify misinformation within user prompts, we utilize GPT-4
as a response judge."
Is it assumed that GPT-4 can 100% identify whether the response of an LLM …
-
When I install lighteval form main and call this command:
```bash
accelerate launch --num_processes=1 run_evals_accelerate.py \
--model_args "vonjack/Phi-3-mini-4k-instruct-LLaMAfied" \
…
-
### Checked other resources
- [X] I added a very descriptive title to this issue.
- [X] I searched the [LangGraph](https://langchain-ai.github.io/langgraph/)/LangChain documentation with the integ…
-
**What problem or use case are you trying to solve?**
Gemini has some very high safety settings by default, which cause it to refuse to generate code sometimes.
**Describe the UX of the solution…
-
Have trained `update_v2` branch on :
* Extracted Semantic token from HuBert Large layer 16 with 1024 cluster Kmean. (`50 tok/sec`)
* Extracted Acoustic token from Encodec 24 khz sample rate, 240 ho…
-
It's useful to be able to run a quantized transformer exported TorchScript model on CUDA even if some quantized operators are performed by dequant/conversion to float32/requant (some sort of temporary…
-
### 1. Reproduction Steps
Try to change the author/instructor of a lesson with the new editor (Gutenberg). If the WordPress installation has only one administrator and all instructors it does not s…
-
# Add the ability to add human feedback via an API call
In many applications, but even more so for LLM applications, it is important to collect user feedback to understand how your application is p…
-
The image resizing routines in `image` seem to be really slow compared to the competition.
For example, [`fast_image_resize`](https://github.com/cykooz/fast_image_resize) crate [benchmarks](https:/…