[BOUNTY - $1000] Get exo Python node running on iOS

[AlexCheema]

Background

I experimented with a rust-based exo implementation that used UniFFI for foreign language bindings so I could run it from a Swift iOS app. I didn't like this design -- I don't want to maintain two different exo implementations. I want to keep the core exo implementation in 100% Python.

This bounty is for getting the exo Python node software running on iOS. I tried experimenting with Pyto however it didn't work with exo because of dependency issues / use of asyncio. By default Pyto uses Python 3.10 and uses the binary from here however there are other binaries available up to Python 3.13.

For now don't worry about integrating the inference itself into exo. We can figure that out in a subsequent PR (probably unify with #238). For now simply implement a SwiftDummyInferenceEngine which calls some function from Swift which will eventually be replaced with some code that runs with Metal GPU acceleration (e.g. #238).

Deliverables

• A working iOS app that runs the Python exo node implementation
• SwiftDummyInferenceEngine that runs an inference engine that calls some dummy function in Swift.
• A circleci CI/CD pipeline that generates the iOS app
• A circleci CI/CD pipeline that runs an e2e test for the iOS app

Future Directions

My vision eventually is to unify this with #238 so that we can generate an InferenceEngine that can run models on iOS with GPU acceleration using tinygrad. Another option is to manually write a small InferenceEngine in swift that uses MLX Swift.

[Yvictor]

Have you tried using Tauri v2 for your Rust-based exo implementation? Tauri is great for creating cross-platform apps, including iOS, and it allows you to keep most of your logic in Rust while providing support for Swift bindings without maintaining two different implementations. Tauri also offers good CI/CD support, which could help with your pipeline setup on CircleCI. Rust enables the deployment of lightweight binaries, making it faster and more efficient.

https://v2.tauri.app/

If you have a Rust version of exo, I can help with integrating it into a Tauri-based app. Using Rust for the core of exo would not only simplify maintaining a single implementation, but it would also make it easier to run exo on WebAssembly (WASM). #304 candle is already using WASM and can run in the browser.

I've already tried using Tauri v2 to build a uv sidecar version exo-app. If with rust based exo, it would be much easier.

[AlexCheema]

Have you tried using Tauri v2 for your Rust-based exo implementation? Tauri is great for creating cross-platform apps, including iOS, and it allows you to keep most of your logic in Rust while providing support for Swift bindings without maintaining two different implementations. Tauri also offers good CI/CD support, which could help with your pipeline setup on CircleCI. Rust enables the deployment of lightweight binaries, making it faster and more efficient.

https://v2.tauri.app/

If you have a Rust version of exo, I can help with integrating it into a Tauri-based app. Using Rust for the core of exo would not only simplify maintaining a single implementation, but it would also make it easier to run exo on WebAssembly (WASM). #304 candle is already using WASM and can run in the browser.

I've already tried using Tauri v2 to build a uv sidecar version exo-app. If with rust based exo, it would be much easier.

I haven't tried it but I'd really like to keep exo core in Python. As much as I personally like rust, I don't think it's the right move for the project -- the ML ecosystem is 99% Python, both in devs and libraries/frameworks.

[shivance]

hey @AlexCheema I want to take a crack at this!

[ToSeven]

This work is perfect for me. I am familiar with Flutter, Dart, Python, and LLM inference tech. Flutter is an open-source framework for building beautiful, natively compiled, multi-platform applications(windows, Mac, Linux, iOS, and Android) from a single codebase. It has better performance and compatibility than Tauri. We can use Nuitka3 to package Python code into a C++ library, which can be called in Flutter with hybrid development technology. @AlexCheema

[shivance]

okay, this is an interesting challenge @AlexCheema. Here is what I have tried so far:

Since you said that you ran into problems while running exo on Pyto, because of asyncio version, I decided to experiment with Beeware . so I can write my app in Python and run it on iOS. after 3 hours of effort, I couldn't get Beeware running. the project is very buggy and their examples aren't running on a cleanly installed environment.

Then, I did some research on Python code calling from Swift, and found that Pythonkit https://github.com/pvieito/PythonKit is my guy. I set up a plain Xcode project for an iOS App that does the addition of two numpy arrays.

( I am using iPad Air 5th gen and iPhone 15 pro for development here )

however python kit is having trouble finding python library

Loading symbol 'Py_Initialize' from the Python library...
Trying to load the library at '/Users/anshuman/.pyenv/versions/3.12.4/lib/libpython3.12.dylib'...
PythonKit/PythonLibrary.swift:59: Fatal error: 'try!' expression unexpectedly raised an error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.

I re-read README and they mentioned that something like this might happen, and for that, you have to set environment variables. However, that didn't work either when I set up all Python-related environment variables on Xcode.

I did some more research on Python calling from Swift:

1. https://docs.python.org/3/extending/embedding.html#embedding

I think running Python code as it is might just be hard and bug-prone. I also suspect some performance tradeoffs here.

We should consider something like mlx-swift as the backend.

[AlexCheema]

okay, this is an interesting challenge @AlexCheema. Here is what I have tried so far:

Since you said that you ran into problems while running exo on Pyto, because of asyncio version, I decided to experiment with Beeware . so I can write my app in Python and run it on iOS. after 3 hours of effort, I couldn't get Beeware running. the project is very buggy and their examples aren't running on a cleanly installed environment.

Then, I did some research on Python code calling from Swift, and found that Pythonkit https://github.com/pvieito/PythonKit is my guy. I set up a plain Xcode project for an iOS App that does the addition of two numpy arrays.

( I am using iPad Air 5th gen and iPhone 15 pro for development here )

however python kit is having trouble finding python library

Loading symbol 'Py_Initialize' from the Python library...
Trying to load the library at '/Users/anshuman/.pyenv/versions/3.12.4/lib/libpython3.12.dylib'...
PythonKit/PythonLibrary.swift:59: Fatal error: 'try!' expression unexpectedly raised an error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.

I re-read README and they mentioned that something like this might happen, and for that, you have to set environment variables. However, that didn't work either when I set up all Python-related environment variables on Xcode.

I did some more research on Python calling from Swift:

1. https://docs.python.org/3/extending/embedding.html#embedding

I think running Python code as it is might just be hard and bug-prone. I also suspect some performance tradeoffs here.

We should consider something like mlx-swift as the backend.

Thanks for posting your findings here. I think this is still do-able however it might require a lot of changes e.g. removing certain dependencies, that's why it's a $1000 bounty.

[AlexCheema]

This work is perfect for me. I am familiar with Flutter, Dart, Python, and LLM inference tech. Flutter is an open-source framework for building beautiful, natively compiled, multi-platform applications(windows, Mac, Linux, iOS, and Android) from a single codebase. It has better performance and compatibility than Tauri. We can use Nuitka3 to package Python code into a C++ library, which can be called in Flutter with hybrid development technology. @AlexCheema

Sounds perfect. Assigned!

Be sure to check out the work being done on #238 and #354.

[ToSeven]

I am waiting for the progress of #238 and #354