search

cognitivecomputations / spectrum

Apache License 2.0
91 stars 14 forks source link

regarding evaluation commands #8

Closed sorobedio closed 1 month ago

sorobedio commented 1 month ago

Hello,

I would like to ask a few questions regarding the evaluation of the Llama-Instruct model. Specifically:

  1. Do you set use_chat_template and use multi-turn few-shot prompts during evaluation?
  2. Is flash attention utilized if possible in your setup?

Additionally, I am working on reproducing the results of the VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct, which I believe is based on your work. Could you clarify if the work I mentioned here is fine-tuned using LoRA, or if it's utilizing a combination of Spectrum and QLoRA?

Thank you for your time and assistance.

dgolchin commented 1 month ago

Hi @sorobedio,

  1. we have evaluated the model according to the specifications of huggingface. I.e. the lm eval harness version that hf uses for its leaderboard with chat template true.
  2. without flash attn

You should find all further answers to your questions in the modelcard: https://huggingface.co/VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct We used neither qlora nor lora for training but spectrum fft on 25% of the layers.

Regards, David

sorobedio commented 1 month ago

thank you.