search

LAION-AI / Open-Assistant

OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.
https://open-assistant.io
Apache License 2.0
36.94k stars 3.22k forks source link

SFT of LLama 2 #3582

Closed gsaivinay closed 1 year ago

gsaivinay commented 1 year ago

Hello,

Just today, Meta open sourced Llama 2 models. Wondering if OA team is considering these.

andreaskoepf commented 1 year ago

Absolutely. We are already evaluating the models. We will start own fine-tuning runs soon (in a couple of hours).

Billyroot commented 1 year ago

That's good to know.

Le mer. 19 juil. 2023 à 00:49, Andreas Köpf @.***> a écrit :

Absolutely. We are already evaluating the models. We will start own fine-tuning runs soon (in a couple of hours).

— Reply to this email directly, view it on GitHub https://github.com/LAION-AI/Open-Assistant/issues/3582#issuecomment-1641087987, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACVZJTUM2EH3LVCVXGH7TO3XQ4HH5ANCNFSM6AAAAAA2O4RWHI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

gsaivinay commented 1 year ago

We will start own fine-tuning runs soon (in a couple of hours).

This is really great. Please keep us posted if possible, on the training and evaluation progress.

flozi00 commented 1 year ago

Is there any place to track the training ? I running some experiments using lora, no resized embeddings for OA special tokens, sequence length of 4096 tokens The loss of 13b model is stable and lower than 2048 tokens falcon 7b

gsaivinay commented 1 year ago

Is there any place to track the training ?

I'm assuming the training progress will show up here as soon as the process is started, but I might be wrong.

andreaskoepf commented 1 year ago

I'm assuming the training progress will show up here as soon as the process is started, but I might be wrong.

We publish/copy successful runs afterwards to the public-sft wandb project.

Billyroot commented 1 year ago

So we just have to wait for the release :). Do you think it might happen next week?

gsaivinay commented 1 year ago

well, 13b model is here https://huggingface.co/OpenAssistant/llama2-13b-orca-8k-3319 and it is with 8k context length 🤯

olliestanley commented 1 year ago

Here is the first oasst dataset SFT model for Llama2 70B. As usual details are on the HF page and future models will also be on HF, so I will close this issue now

https://huggingface.co/OpenAssistant/llama2-70b-oasst-sft-v10