-
## 🐛 Tuple in the config dictionary changed to string when run a python notebook with papermill
I used the current config to run papermill with the following script
#### The config dictionary
…
-
#params = 151111638
#non emb params = 41066400
| epoch 1 step 50 | 50 batches | lr 0.06 | ms/batch 1378.43 | loss 7.85 | ppl 2570.784
| epoch 1 step 100 | 100 batches | lr 0…
-
```
When I use interpolate-ngram to interpolate two models by CM or GLI with
perplexity optimization, I get following faults:
1st:
interpolate-ngram -lm "model1.lm, model2.lm" -smoothing ModKN -inte…
-
```
When I use interpolate-ngram to interpolate two models by CM or GLI with
perplexity optimization, I get following faults:
1st:
interpolate-ngram -lm "model1.lm, model2.lm" -smoothing ModKN -inte…
-
I'm trying to run this against a local instance of oobabooga exposing openai-like API:
```
OPENAI_API_KEY=x lm_eval --model local-chat-completions --model_args model=mymodel,base_url=http://127.0.…
-
Hello,
I'm trying to reproduce paper numbers on PopQA by running the following command :
Question Answering
python run_short_form.py \
--model_name selfrag/selfrag_llama2_7b \
--input_file ev…
-
If you cmaim that GPQA task has no subtask which task is used by the leader board.
here are the list of tasks associated with the gpqa task in lm-eval
| Tasks |Version|Fil…
-
How can we integrate the use of language models to evaluate language model generations?
Currently, lm-eval evaluates language model generations with conventional metrics such as accuracy, bleu, etc…
-
Hey,
when I run "compile-graph.sh" I run into the following error:
ngram-count -wbdiscount -order 4 -text db/extra.txt -lm data/extra.lm.gz
+ ngram -order 4 -lm db/en-230k-0.5.lm.gz -mix-lm d…
-
**Environment Variables for Initialization:**
I recommend using environment variables to initialize the following parameters when using lm-studio:
API_URL: The URL for the API you’re interacting wi…