search

evilsocket / cake

Distributed LLM and StableDiffusion inference for mobile, desktop and server.
Other
2.44k stars 127 forks source link

Req Support for Llama 3.1 #21

Open jkfnc opened 1 month ago

jkfnc commented 1 month ago

Llama 3.1 got released, anything needs to change to get it working.

evilsocket commented 1 month ago

is this a question, a statement or what... ?

jkfnc commented 1 month ago

Sorry, its a question, since LLama 3.1 needs RoPE scaling is there any changes needed to Cake for it to work .

evilsocket commented 1 month ago

RoPE is implemented here https://github.com/evilsocket/cake/blob/8a782b8a76f5ae1c3438c0853c9ecd34306d4197/cake-core/src/models/llama3/attention.rs#L25 ... to be honest i didn't have time yet to check the structural differences with LLama 3.1