search

tijiang13 / InstantAvatar

333 stars 23 forks source link

Could the code run on multi gpus? #42

Closed yf1019 closed 10 months ago

yf1019 commented 11 months ago

Hi tianjian,

Really appreciate your work as Fast-Snarf and InstantAvatar. I'm trying to involve this module into my project but I found it failed to run on the multi gpus. May I ask how can I use it (especially fast-snarf) on multi gpus setting?

Thank you so much.

tijiang13 commented 11 months ago

Hi Yangfan,

We haven't tested multi-gpu training but I guess it will not be too difficult as each ray is handled in parallel. But to do that you probably need to correct some .cuda() or .to(device) in the code which may cause some troubles.

Best, Tianjian

yf1019 commented 11 months ago

Hi Thanks for your reply.

However, I test the multi-gpu setting and carefully check the tensor device. It still gives error like cuda memory faulty on fast-snarf. I'm not sure the reason.

tijiang13 commented 11 months ago

That's weird. Usually illegal memory accesses are caused by tensors originating from different devices (if it worked on single GPU). However, if that possibility has already been ruled out, then it's difficult for me to pinpoint the cause.

Best, Tianjian

yf1019 commented 11 months ago

Hi Tianjian,

No problem. Thanks so much for your help.