search

OpenBMB / Eurus

Apache License 2.0
262 stars 15 forks source link

Has there been a mix up with the files uploaded to HF for the 70b models? #3

Closed jukofyork closed 4 months ago

jukofyork commented 4 months ago

Has there been a mix up with the files uploaded to HF and some of the CodeLlama-70b-Instruct-hf files have been used instead of CodeLlama-70b-hf:

https://huggingface.co/openbmb/Eurus-70b-nca/discussions/3

None of the float types seem to match, the chat template is the strange one used by CodeLlama-70b-Instruct (including the <step> token), and the ROPE frequency and context length both match CodeLlama-70b-Instruct rather than CodeLlama-70b.

But the appendix of the paper quite clearly implies Eurus-70b was trained on top of the base CodeLlama-70b:

On MMLU, EURUS outperforms baselines dedicated to coding and math, and achieves higher results than Mistral-Instruct-v0.2 and CodeLLaMA-70B-Instruct, the official aligned versions of our base model built by their authors.

I'm quite interested to try the model as it's the first fine-tuned model of CodeLlama-70b other than Phind-70b (which is private/proprietary). I'm downloading the Safetensors data now and going to try copying the rest of the files from the original CodeLlama-70b-hf and then edit in the suggested [INST] <prompt> [\INST] chat template to see if it works.

There is already once person who has quantized a GGUF from this possibly corrupt upload so if there has been a mix up then it would be best tell them to take it down! :)

cgq15 commented 4 months ago

We fine-tune CodeLLaMA-70B-base with the same 4k/10k settings as CodeLLaMA-70B-Instruct.

We used the prompt template described in the model card and the dtype is fp16, but we used the config file of CodeLLaMA-70B-Instruct. Sorry for the ambiguity. We have fixed this issue on huggingface.

jukofyork commented 4 months ago

Closing this as it all seems fixed on HF now.