search

evilsocket / cake

Distributed LLM and StableDiffusion inference for mobile, desktop and server.
Other
2.57k stars 135 forks source link

Questions for Xcode setup for testing/dev #4

Closed chadbrewbaker closed 3 months ago

chadbrewbaker commented 3 months ago

Taking a quick look:

https://github.com/evilsocket/llama3-cake/blob/main/cake-ios/src/lib.rs is the secret sauce for iOS?

I have a Mac Studio. Can I use the iOS simulator to simulate a few devices for testing this, if so how?

At a high level how is the model split between devices? Would it be possible to write a WebGPU client where you plug in the IP address, ssh creds, and it becomes a worker node?

evilsocket commented 3 months ago

https://github.com/evilsocket/llama3-cake/blob/main/cake-ios/src/lib.rs is the secret sauce for iOS?

There's no secret sauce really, rust is awesome and can easily be compiled for anything. The cake-ios crate is just a small wrapper that basically is the equivalent of main in cli.

I have a Mac Studio. Can I use the iOS simulator to simulate a few devices for testing this, if so how?

I wouldn't use a simulator for this, you need real hardware otherwise the worker is just too slow.

At a high level how is the model split between devices?

You decide! That is what the topology is. You can check the README that'll show an example of how to distribute the layers of llama3.

Would it be possible to write a WebGPU client where you plug in the IP address, ssh creds, and it becomes a worker node?

Yes, as the README clearly states. SSH has nothing to do with it tho.