search

facebookresearch / spiritlm

Inference code for the paper "Spirit-LM Interleaved Spoken and Written Language Model".
Other
747 stars 47 forks source link

Would consider release the training code? #15

Open luohao123 opened 1 week ago

luohao123 commented 1 week ago

Would consider release the training code?

tuanh208 commented 1 week ago

Hi, we don't have any plans to release the training code, but I think you can use our speech tokenizer to fine-tune (or continue pre-training) on any llms and it should work

luohao123 commented 1 week ago

Hi, the speech tokenizer seems can be used directly, however, when using a new LLM, how could cooperate the LLM and components to get a workable result?

tuanh208 commented 6 days ago

I think the steps would be 1) Collect some aligned speech-text data 2) Tokenize the speech-text data in an interleaved fashion (using speech tokenizer) 3) Extend llm vocab and continue llm training on 2) with data collected from 1)