Apple Silicon - metal, mps, mlx support, m1, m2, m3, m4

matatonic / openedai-vision

An OpenAI API compatible API for chat with image input and questions about the images. aka Multimodal.

GNU Affero General Public License v3.0

179 stars 14 forks source link

Open doreilly257 opened 3 weeks ago

doreilly257 commented 3 weeks ago

Any chance you can include the capability to run on Apple Silicon M1/M2/M3/M4 processors with metal, mps, or mlx support.

matatonic commented 3 weeks ago

Perhaps, there are a lot of 'cuda' assumptions in many models, but I'll try to make it possible.

matatonic commented 2 weeks ago

Currently device=mps is possible, and if it's available it will automatically select it. Which model are you trying to use?

yukiarimo commented 2 weeks ago

Does it works with Pixtral and other Vision models on MacOS?

matatonic commented 2 weeks ago

Does it works with Pixtral and other Vision models on MacOS?

Sorry, but I don't know, I don't have an apple device to test with.

Has anyone else tried?