-
### Please check that this issue hasn't been reported before.
- [X] I searched previous [Bug Reports](https://github.com/OpenAccess-AI-Collective/axolotl/labels/bug) didn't find any similar reports.
…
-
Hello everyone, thank you for contributions so far, I've been working through them and these tasks are forming a challenging a comprehensive benchmark for modern LLMs and LLM programs. We worked on [C…
-
**Is your feature request related to a problem? Please describe.**
I can't modify the existing template + rails on the evaluator object to customize for my use case.
**Describe the solution you'd…
-
Batch view / definition routes
**7 Routes**
---
**POST** /evals/batch
_accessible to all upperclassmen_
Creates a new batch with either specified criteria or specific members
If created by the eval…
-
### Problem Description
Hi, everyone! I check the code of function `sampling_estimate`.
Assume we have a data instance `x` with `M` features.
- We keep the 1st to j-th feature as original, replace …
-
### Short description and motivation for the proposed feature
Idea from @Ricram2: doing `EVALUATE * FROM` would yield a table with all compatible accuracy metrics for the model being evaluated.
### …
-
Perplexity to start
-
Thanks for your brilliant work! Having downloaded K400 pretrained checkpoint file(k400-probe.pth.tar) and modified the config yaml file for the corresponding dataset(specifying datapath), I ran evals.…
-
I have been trying to extract data (title, question answered, entities, summary) from documents chunks.
I believed typed predictors would be good for this, but I keep running into "Too many retrie…
-
The following is posted verbatim from @dtkerrs review of #49 with regard to the `switchy.distribute.MultiEval` interface and implementation:
> So some feedback here on `evals` is that it might be ni…