search

artidoro / qlora

QLoRA: Efficient Finetuning of Quantized LLMs
https://arxiv.org/abs/2305.14314
MIT License
9.74k stars 800 forks source link

Can you help implement MMLU evals into Axolotl? #248

Open teknium1 opened 11 months ago

teknium1 commented 11 months ago

https://github.com/OpenAccess-AI-Collective/axolotl/pull/441


    return self.call_event("on_evaluate", args, state, control, metrics=metrics)
  File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer_callback.py", line 406, in call_event
    result = getattr(callback, event)(
  File "/home/teknium/evaltest/axolotl/src/axolotl/utils/callbacks.py", line 166, in on_evaluate
    (loss, logits, labels) = trainer.prediction_step(
  File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3329, in prediction_step
    has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)
  File "/home/teknium/evaltest/axolotl/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3329, in <genexpr>
    has_labels = False if len(self.label_names) == 0 else all(inputs.get(k) is not None for k in self.label_names)
AttributeError: 'NoneType' object has no attribute 'get'