search

hyunwoongko / openchat

OpenChat: Easy to use opensource chatting framework via neural networks
Apache License 2.0
438 stars 57 forks source link

More GPU Support #37

Closed ibivibiv closed 3 years ago

ibivibiv commented 3 years ago

I found the rest of the locations where the device was not getting set to be a gpu id.

The most significantly new location was in the base parlai.py agent. @104 you can see where I had to check if the device is a gpu and then set the batch to be assigned to the gpu. If not the batchify call creates a default assignment to the cpu and the model is on the gpu so the entire prediction fails because they are on different devices. I have also created some examples of using gpu to make it clear to a user how to use the gpu parameter. The only pending use case would be if there were multiple gpu's in a system. I am not quite sure how to support that just yet. As is the GPT XXL and the other XXL models probably won't run for most people as they use more memory than even a V100 provides on a single GPU. They will require multiple cards. I will try to figure out how to accommodate this later, if I get a chance to test on a system that has more then one card.

hyunwoongko commented 3 years ago

Thanks a lot !

Thank you for making a lot of contributions.

hyunwoongko commented 3 years ago

I'll be deploy the version you made soon.