Is it possible to support MPI?
It's implemented in upstream but I didn't see any traces of it working here.
Specifically it will allow spliting larger model into parts that could be individually loaded into each workers individual gpu's and then accelerate greatly token generation of larger models. Have anyone tested it with koboldcpp and is it possible to run?
Multi device inference is probably out of scope for KoboldCpp, but multi GPU inference is possible in CuBLAS mode, you can configure with --tensor_split
Is it possible to support MPI? It's implemented in upstream but I didn't see any traces of it working here. Specifically it will allow spliting larger model into parts that could be individually loaded into each workers individual gpu's and then accelerate greatly token generation of larger models. Have anyone tested it with koboldcpp and is it possible to run?