JohannesGaessler
commented
10 months ago See the conversation starting at https://github.com/ggerganov/llama.cpp/pull/3776#issuecomment-1782868497 . I am aware of the parallelization scheme where the model is split into blocks of layers instead of splitting each layer into slices. As I said before: I have no intention of implementing it. Multi GPU only really makes sense for running something like 70b and for that purpose I think the best buys are either multiple P40s or multiple RTX 3090s. For multiple P40s the current scheme works better while for multiple RTX 3090s NVLink is available which should also result in low parallelization overhead. Synchronization overhead may also vary by OS: if you e.g. use Windows peer access between devices is only available via NVLink so the performance for multiple GPUs working on small batches should be worse.
This means the calculations can all be computed without synchronization on a single GPU at highest possible performance of that GPU.
No, for $N$ identical GPUs serving one request the maximum theoretical GPU utilization using that scheme is $\frac{1}{N}$ because the GPUs have to wait for each other. The only way to achieve 100% GPU utilization would be to serve multiple concurrent requests (for example by serving multiple users) in parallel.
Also: see https://github.com/ggerganov/llama.cpp/pull/3814 and check whether that PR has unintentionally resulted in a performance regression for your hardware setup.
cmp-nct
jezzarax
KerfuffleV2
ggerganov
Ph0rk0z
Problem: I am aware everyone has different results, in my case I am running llama.cpp on a 4090 primary and a 3090 secondary, so both are quite capable cards for llms. I am getting around 800% slowdowns when using both cards on the same model and settings (basically regardless which model I tried), batch processing speed can go down from 2400t/sec to 200-300t/sec (8-10 times slower than on single GPU). This happens as soon as any tiny bit of processing (-ts) is shifted to the 2nd card.
I assume it is a synchronization problem in the cuda loops, I also assume the issue does not affect every combination of GPUs, especially if one GPU is significantly slower.
Suggestion: My suggestion is to add a parameter like -layer-split, when this is used the tensors are not split up, instead the layers are split into the cards (using -ls instead of -ts). This means the calculations can all be computed without synchronization on a single GPU at highest possible performance of that GPU.
Caveat: In theory tensor split should boost performance, as both cards can process a split tensor at the same time, so it's the better solution but currently that's so far from reality, the suggested layer split should significantly boost the processing speed.
@JohannesGaessler what do you think ?