search

mybigday / llama.rn

React Native binding of llama.cpp
MIT License
300 stars 24 forks source link

OpenCL Implementation for Android #40

Closed Vali-98 closed 9 months ago

Vali-98 commented 9 months ago

First of all, thanks for the hard work on bringing this project to the react-native ecosystem.

I have been using llama.rn for a few weeks now in my personal project: https://github.com/Vali-98/ChatterUI

I was wondering if there is any interest in implementing OpenCL for android. I have attempted to work on it myself to little success, given my inexperience with native modules.

jhen0409 commented 9 months ago

In my understanding, not all Android devices supported OpenCL, for example the Google Pixel does not.

I think a better solution is to use Vulkan, which llama.cpp should support soon. But I think their support for iGPU is not enough (currently Vulkan features for Android devices are usually limited).

Vali-98 commented 9 months ago

Though I would prefer a mature solution in OpenCL, Vulkan does seem to be a bit more future proofed. Perhaps this is simply some impatience from my end to have improved prompt processing, but Vulkan does seem to be a still far out. As it stands, some users for my project would also like to see improved processing on Android.

Granted, I am not the repo maintainer so your perspective on implementation is far more valid. I suppose I will continue working on a forked version to get opencl functional, thanks for the response.

Ar57m commented 9 months ago

vulkan is now merged, is it possible to bring for android as an option?

Vali-98 commented 9 months ago

Likely not yet, I've heard word that many mobile GPUs still are not functional.

I will be closing this issue, as now its simply waiting until vulkan's mobile implementation is done.