search

conceptofmind / LaMDA-rlhf-pytorch

Open-source pre-training implementation of Google's LaMDA in PyTorch. Adding RLHF similar to ChatGPT.
MIT License
466 stars 76 forks source link

Pre Trained Model? #8

Open OfficiallyMelon opened 1 year ago

OfficiallyMelon commented 1 year ago

Could you upload a pre trained model for those with less powerful GPUs, You could upload to Hugging face. Thanks. I just want a ChatGPT Alternative that can run locally and have a chat with the model (like in the image) Thanks

conceptofmind commented 1 year ago

Firstly, it costs ~$20,000 to $80,000 to train a small foundational model. But that is the plan if I can get funding. Although I need to finish collecting both datasets first that would be required to do such. And this would require three stages of fine-tuning for instruct and dialog. I promised lucidrains I would first train a PaLM model for the PaLM-rlhf-pytorch repository: https://github.com/lucidrains/PaLM-rlhf-pytorch