-
while trying to tun the synthesizer for generate_goldens_from_docs as per the documentation https://docs.confident-ai.com/docs/evaluation-datasets-synthetic-data#creating-an-synthesizer, facing an iss…
-
`ilab model train` has convenient logic to loop through a checkpoints directory and return the best-score & best-model from a list of candidate models. (https://github.com/instructlab/instructlab/blo…
-
I create a subclass of baseragassembeddings. because I already have all the embeddings for context, query, and question. I did this to not use the openai API key. because it is costly and also I want …
-
Reproduction:
```
lm_eval --model hf-multimodal \
--model_args pretrained=llava-hf/llava-1.5-7b-hf,max_images=1 \
--tasks mmmu_val \
--device cuda:0 \
--batch_size 8
```
Erro…
-
We should have benchmark folder where we add all benchmarking models that work on real-world datasets. For those models we want to track end-to-end performance and evaluation score. Also, we want to a…
-
It is a interesting work. When I just evaluate the pretrained model from the author provided, I get the lower results:
![image](https://github.com/FoundationVision/GenerateU/assets/57434913/9e56c85f-…
-
### Search before asking
- [X] I have searched the Ultralytics YOLO [issues](https://github.com/ultralytics/ultralytics/issues) and [discussions](https://github.com/ultralytics/ultralytics/discussion…
-
### Model ID
tiiuae/falcon-11B
### Model type
Decoder model (e.g., GPT)
### Model languages
- [X] Danish
- [X] Swedish
- [X] Norwegian (Bokmål or Nynorsk)
- [X] Icelandic
- [X] Faroese
- [X] Germ…
-
hello, i m impressed by the Decompile model you released.
i want to know the details of splitting exebench for train and evaluation. because i want to reproduce your evaluation results for a bette…
-
### Model ID
google/madlad400-8b-lm
### Model type
Decoder model (e.g., GPT)
### Model languages
- [X] Danish
- [X] Swedish
- [X] Norwegian (Bokmål or Nynorsk)
- [X] Icelandic
- [X] Faroese
- [X]…