Open INF800 opened 5 months ago
Can we do distributed inference with multiple GPUs? i.e Loading parts of a model onto each GPU.
Yes, it is possible. We can split the model across more than one GPU if it doesn't fit in one. The current version of the library does not support it, but I will add that when I have more bandwidth.
Is there any method available out of the box to run
FactChecker("hf:kundank/genaudit-usb-flanul2")
inference with multiple GPUs? My 24gigs GPU goes OOM.