search

kyutai-labs / moshi

Apache License 2.0
6.74k stars 524 forks source link

Request to Open-Source Helium Base Model #117

Closed ArmeriaWang closed 1 month ago

ArmeriaWang commented 1 month ago

Hi moshi team,

Thank you for the great work on moshi! Would it be possible to open-source the Helium base model? As a single-language large model, it has significant potential for fine-tuning on speech tasks and would be a valuable resource for both the speech and NLP communities.

Thanks for considering!

adefossez commented 1 month ago

We do not have plans to release the text model Helium at this time. There is already a large number of open source text models that are likely to be good fits. Or maybe you mean the base Moshi model ? Similarly, we have no plan to release it at the moment.

Ronnie-Leon76 commented 1 month ago

@adefossez Is it possible to swap Helium, 7B-parameter text LLM with Llama 3?

LaurentMazare commented 1 month ago

The approach we use for moshi consist in training the 7b model on speech data with an initialization based on text data, so the cost of retraining on another LLM is significant. Also the llama-3 license is not very permissive so if we were to do this we would have to call the model llama-moshi so we don't have some immediate plans for doing this.