search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
10.64k stars 1.06k forks source link

Is there any support for visual generation? #1606

Open dunbar12138 opened 3 months ago

dunbar12138 commented 3 months ago

Hi, thanks for the great repo.

I'm wondering how I can use the Lit Autoregressive models in the image/video generation tasks.

Is there any doc / tutorial about how to prepare visual tokens as training set, and how to make the LitGPT/llama model take in tokens instead of text?

Or even better, is there any tutorial / example about how to integrate LitGPT/llama into a normal pytorch lightning training recipe (potentially with other models getting trained together), and customize the architecture and training instead of being a standalone model itself?

rasbt commented 3 months ago

Hi there!

Image/video is not supported but I can surely add a Trainer recipe some time. Thanks for suggesting!

dunbar12138 commented 3 months ago

Thanks for the reply! That would be great. Let me know when the tutorial is added. I really appreciate it!