search

huggingface / candle

Minimalist ML framework for Rust
Apache License 2.0
13.48k stars 724 forks source link

Metal iOS #1841

Open soldelacroix opened 1 month ago

soldelacroix commented 1 month ago

Great framework!

Is the usage of Metal already possible on iOS? I'm trying to run the Phi example on iOS and I can only get it to work with a CPU device but not with Metal. MTLStorageModeManaged isn't available on iOS.

LaurentMazare commented 1 month ago

I've never tried compiling for iOS but this issue seems related #1759

ssoudan commented 1 month ago

Have had a try at it here but something based on candle-examples/examples/phi is generating mostly blank tokens on my iPad. Not sure what the issue is at this point. TBC

filipw commented 2 weeks ago

Thanks a lot @LaurentMazare for the great work.

I tried running it on iOS and as the OP noted, it works well but only on CPU. When using the GPU, the Metal code crashes because candle is explicitly using MTLResourceOptions::StorageModeManaged in a few places. The managed mode is not available on iOS and tvOS so the code panics.

I tried simply changing it to be shared everywhere, but that also panics because iOS has a buffer size limit of 256MB (see: https://github.com/gfx-rs/metal-rs/blob/master/src/device.rs#L712-L718) and at least trying to run Phi would attempt to allocate more.

Can we somewhere control the buffer size that is being used by candle?