Multi-GPU loading of models produces gibberish

[nktice]

Here's another bug on Oobabooga's project that is unresolved... https://github.com/oobabooga/text-generation-webui/issues/2923 I realized that the ExLlama team may have a solution.... So I posted this there, and a maintainer replied, the problem's with HIP - https://github.com/turboderp/exllama/issues/281 So I'm writing here in the hopes that someone will resolve the issue.

Here's the guide I wrote to get everything working on AMD kit... https://github.com/nktice/AMD-AI Models load fine when it is only on one card, here are some results : https://github.com/nktice/AMD-AI/blob/main/SallyAIRiddle.md

Multi-card loading only spits out gibberish, here's an example :

pha golden Riv. Jcatred (ProcSN proc Dre -:// Mindly means for the and in a Nich říct Forest Rav Rav fran fran fran gaz Agrcastle castleasiacliordinate advers Mem advers Basibenkooor paste Singapore refugeermeanny intellectualsafe Shakespe contempor Mallmanual Quantmousektr Ge Mil shadownehfdzekADmobile Und Euenf Next Dominbuchcock Infoengo‭ Hann NAT ]] Ferr' -.-- -,-

    ason, rang,-, –-

(,,

--,.,

alter

,-

(

-on,-.

I,- .

1

V

V. film-

N

    –on.,on,.

(, for.

and of- is. . and –on, –,. and

In in

film school and I on and with and I ":

.

` andon util –

[kjayapra-amd]

Do you have a simple HIP Sample where this issue is reproduced?

[nktice]

Here is more info about what the issue was on another thread - https://github.com/ROCm/rocBLAS/issues/1346#issuecomment-1905404797 [ As I mentioned there, at the bottom of that thread - it is now resolved. ] Here's the guide I wrote with the instructions I used and have it working - https://github.com/nktice/AMD-AI/blob/main/ROCm6.0.md