Swift bindings?

[vmanot]

Are Swift bindings to MLX in the roadmap/within scope?

I attempted to wrap up the built library + metallib file into a macOS .framework bundle, giving it a correct .modulemap and importing it into a Swift target with C++ interop enabled, but soon ran into this roadblock:

From Swift's documentation here:

Swift currently cannot import C++ modules introduced in the C++20 language standard.

I understand that C++20 is the blocker here, but I'm wondering if somehow the headers could be made backwards compatible for a version that the Swift compiler can understand.

[awni]

Yes were definitely considering a swift front-end and it's well within scope. Prioritizing it depends on how useful it is to MLX users. If you care to say more about your use case / what you'd be interested in doing with it, we'd love to hear.

[JacopoMangiavacchi]

3 years ago I developed this swift side project to generate and train on device coreml models using Swift result builder DSL approach.

https://github.com/JacopoMangiavacchi/SwiftCoreMLTools

I still believe Swift DSL could be a nice interface to create model architecture in MLX

[vade]

Hi

Swift bindings would be amazing for a few reasons.

Im one of the founders at https://ozu.ai - we're making tools for multi modal creative video understanding which uses inference locally on Apple Silicon

Today, we're using CoreML and some custom Metal kernels to do our analysis along side some other cross platform inference runtimes which dont fully take advantage of

• Unified memory / IOSurface
• Hardware accelerated video encoding and decoding
• ANE
• Metal

Since they sort of assume a dual memory (GPU / accelerator and main memory paradigm).

Secondly, we do all of our research in Python, but ship most of our ML Code in Swift (And C / C++ if we are forced to).

Having a common array language which runs in both languages would make porting, testing and implementing and shipping our Pythonic research code in Swift natively a breeze, which has honestly really been a barrier.

Also, there's some side effects which I mentioned in another thread (https://github.com/ml-explore/mlx/issues/18#issuecomment-1846191659) which would make MLX really interesting for working along side the ANE on inference tasks.

I think it's the right move, from a ton of perspectives. You'd get a subset of swift developers who are eager to leverage ML tooling using the python tooling for rapid prototyping. You'd get a subset of python developers helping ship native apps with better numerical performance and with easier porting of the code.

Its a win win.

Thanks for considering it.

[dougdew64]

My use case is perhaps a bit less important than some others, but I am hoping to create a SwiftUI-based LLM visualization app kind of like https://bbycroft.net/llm but atop mlx.

I had begun my effort atop llama.cpp but would like instead to build atop mlx because I am likely to find more support from Xcode users in the mlx community. For example, I've experienced problems getting Xcode shader profiling to work and I have been unable to find anybody else in the llama.cpp community who admits to having any experience with using Xcode shader profiling.

I'm ultimately hoping to add to my LLM visualization some written discussion of LLM code, and of profiling that code, including the use of Instruments.

I suggest that my use case is perhaps less important as I am working on this as a personal project purely for fun.

[jordibruin]

I would love to implement this into www.macwhisper.com to show off Apple Silicon performance even more

[dougdew64]

@vmanot I'm attempting to implement a swift-based version of the c++ tutorial app. I'm new to swift <--> c++ integration and am getting my butt kicked.

For example, my very first line of the swift tutorial is failing to compile. And, I've spent this entire day getting to just this far.

In this screenshot, the mlx project is a build of the mlx repo, generated from the cmake lists in the mlx repo. The tutorial project is the c++ tutorial project in the mlx repo ( I added it to the workspace simply to confirm that I had correctly built the mlx repo ). The mlx.swift stuff is simply a target for providing the swift <--> c++ bridging header. The tutorial.swift stuff is my attempt to re-implement the c++ tutorial in swift.

The Xcode intellisense believes that there's an mlx namespace, and a core namespace and array member of that namespace. But, the compiler doesn't believe any of that.

I'm hoping that you might easily recognize my mistake.

[dougdew64]

I made some changes to names of groups and targets (e.g. eliminated . from names such as tutorial.swift). Still no success. And now, I'm getting a different error.

Everything in Xcode worked very simply, quickly and predictably when I was only using c++. But, since the moment that I began attempting to use swift, I've been blocked by problems which make no sense and at times seem random.

I'm merely grasping at straws at this point, which is a very poor use of time.

I think that I will enjoy writing some c++ for higher level functionality such as NN layers and wait for somebody else to figure out this swift <--> c++ integration.

[vmanot]

@dougdew64 unfortunately I don't think Swift can interoperate with MLX as it currently stands, hence this question when I opened this issue:

I understand that C++20 is the blocker here, but I'm wondering if somehow the headers could be made backwards compatible for a version that the Swift compiler can understand.

C++20 support seems to be partially implemented which is it why it allows you to get as far as you did, but it's officially unsupported as per the language docs.

[dougdew64]

Thanks @vmanot. Your earlier post on this matter is what motivated me to make an attempt today to see what I could accomplish. I had grepped in the mlx code for mentions of "module" and didn't find any, or at least any that seemed like I couldn't overcome.

However, by this afternoon I realized that I was getting my butt kicked by Xcode, Swift<-->C++ details and most of all, my ignorance on this matter.

If the situation is as you've described, and C++20 support is the primary blocker, then that probably has implications for the best strategy for implementing higher level functionality such as NN layers in a way that will ultimately support a Swift interface. Rather than attempt to implement that higher level functionality in Swift, it seems prudent to instead implement that in C++ while awaiting the C++20 support which would enable a thin layering of Swift atop that new C++ functionality.

All of that said, this problem will possibly repeat with C++23 (and future versions).

[douglasdew]

@vmanot What are your thoughts on going old-school and simply building the swift stuff atop an objc wrapper around the mlx c++? My understanding is that swift <--> objc integration is mature and works well.

[vmanot]

@dougdew64 @douglasdew (am assuming you're both the same person here haha), I don't think an ObjC wrapper around the MLX C++ code, further wrapped by another Swift layer would be bad per se, but I don't think the effort would be worth it.

I think it'd also take great care and effort to ensure that no performance is being lost due to the bridging overhead, which again I don't think is something that's a technical impossibility but rather just an extremely tedious and error-prone exercise.

I'm interested in exploring writing a C++17 compatibility layer on top of MLX's C++20 interface, bridging that to Swift and working from there. Since you're motivated enough to try wrangling this, if you'd like to take a crack at this together I'm happy to connect off GitHub and discuss the various strategies I'm considering.

[dougdew64]

@vmanot In my opinion, your C++17 idea is better than my ObjC idea.

That said, the "file not found" error which I reported earlier seems not to to be related to a C++ version. Instead, the error seems to be related to the way in which I'm attempting to use Swift <--> C++ integration. In other words, I've experienced some successes and some failures with Swift <--> C++ integration depending upon how I structure my projects. My assumption is that I'm using Swift <--> C++ integration incorrectly, but it's not clear to me what I can do to use the integration correctly and also in a way which would make sense for this project.

Also, as a quick test, I removed C++20 from the supported language dialects so that only C++17 remained. Yet, I received the same "file not found" build error.

Also, I've performed some other tests, such as not #import-ing any of the mlx header files and instead #importing only an empty header file (foo.h) which is co-located with my bridging header file. I even performed that foo.h test without any header search path setting. Always, I've received the header file not found error. That rules out C++20 issues.

I've compared my project to one of the Apple sample Swift <--> C++ integration projects, and have confirmed that I have configured my project exactly as Apple configured their project. That seems to rule out any mistakes on my part, but perhaps I'm overlooking something.

Honestly, this seems to be exactly the kind of circumstance which made me believe that this Apple-focused repo would be super beneficial to work in. My hope is that somebody from Apple will read this thread and point out my mistake or point out that what I'm attempting to do is not yet supported.

[dougdew64]

@vmanot Regarding C++17, what kind of build errors have you received when attempting to build mlx with C++17 specified as the language dialect?

[dougdew64]

Interestingly, if I simplify the tutorialswift project so that it has only a single target (i.e. there's no framework target within tutorialswift) which bridges in the C++ headers, then the header-file-not-found build errors go away.

Unfortunately, there's still the problem that any code references such as mlx.core.array in main.swift fail. I can see in Xcode intellisense that the Swift mlx enum is being created for the C++ mlx namespace. However, in the Xcode intellisense, that enum has no items. After having read the Swift <--> C++ integration docs regarding C++ namespaces, it seems that what is happening is as expected. In other words, there's no suggestion in the docs that nested namespaces such as mlx.core will work. In other words, there's mention in the docs of the Swift <--> C++ integration being able to add a namespace item to the generated Swift mlx enum for the core namespace.

I'll confirm with a test that the failure is as I'm supposing. At the moment, my guess is that nested C++ namespaces such as mlx.core are not supported. Worse, given that the compiler folks have opted to use Swift enums for all of this, it's not clear that nested C++ namespaces will ever work unless the concept of a Swift enum type is extended somehow.

[UPDATE] I tested by adding a struct to the mlx namespace. That failed. So, my supposition was wrong.

[dougdew64]

I tried one final test, and it succeeded.

I added struct Foo { int bar; }; to the mlx/array.h file, but outside of any namespaces. That worked perfectly with Xcode intellisense and the debugger.

So, I have two takeaways and a possible third takeaway:

1. I was only able to make this work by using a project structure which did not include a middle swift api/framework. In other words, my swift app was forced to integrate the mlx c++ files directly, rather than import a Swift API/framework that had integrated the mlx c++ files. Although I am entirely fine with that, other swift app developers might prefer to not have to think about c++, let alone integration of c++ with swift.

2. Despite what the swift documentation claims, I was not able to integrate the c++ namespaces of the mlx project into my swift app.

3. (Possible) when I refactored my code so that swift <--> c++ integration was performed within my swift app, Xcode popped up a dialog which offered to create a bridging header file for me. Prior to refactoring, I had been using a manually generated bridging header file. Although I'm confident that the contents of my manually generated bridging header file were correct, I'm also entirely confident that there could be some sort of build setting for bridging header files which I had failed to set when I manually authored my bridging header file. I'm looking through settings now.

[vmanot]

@vmanot Regarding C++17, what kind of build errors have you received when attempting to build mlx with C++17 specified as the language dialect?

@dougdew64 it's not something that can be fixed by setting a build flag, from what I understand from some of the errors, the MLX headers are making use of C++20 features and that is why the Swift/C++ interop is failing.

To get Xcode to recognize an mlx import in Swift, I had to do a lot of manual header editing and create a .modulemap myself. I ran into the same roadblock, that I was able to get it to acknowledge and print out a referenced bridged type (mlx.core.), but could not actually use it.

There is also additional Swift-facing work pending on the MLX side irregardless of C++20, to make it easier for the Swift compiler to understand how to bridge initializers etc. (Swift lang docs + evolution proposals detail this).

[dougdew64]

@vmanot my goal when asking about errors when changing the dialect setting to C++17 wasn't to solve any of these problems. Rather, it was to generate compilation failures during the build of libmlx.a due to use of C++20 features. In other words, by turning off support for C++20, we could identify exactly where any C++20 features are being used.

[dougdew64]

@vmanot I was about to perform the build setting experiment which I suggested earlier so that I could identify usages of C++20 features, but noticed that the build setting for C++17 had already been in place when I successfully built mlx a few days ago.

Which observations did you make that confirmed that C++20 features are being used in the mlx code? Given that the build is configured for C++17, it seems that there should be compilation errors if usages of any C++20 features exist in the mlx code.

[xnorai]

It'd be very cool to have access to the full API, similar to PyTorch C++ bindings. MLX is in a unique position here - Swift offers much lower barrier to entry than C++, and better developer workflow scalability (thanks to type checking) than Python. It also natively supports concurrency, and the story is particularly clean here on Apple platforms thanks to GCD. Readability is also quite good. While the attempt to offer Swift FE for TensorFlow failed, this seems like a shoo-in on Apple platforms where Swift not only has gained a foothold, but is often the preferred choice for developers. For this reason I wouldn't limit it to just iOS/iPadOS.

[vmanot]

@dougdew64 it was the Swift compiler citing C++20 in an error, I'll have to retry my setup and get back to you on this soon. Haven't had the bandwidth to work on this during the week, but am going to get back to pushing on MLX <> Swift this coming weekend :D

[Quantisan]

@awni this would save a few months of our development time at least. Thanks for your team's work!

Our use case is a social welfare co-pilot app for frontline social workers. This is for a new non-profit that we're building. Thus, we are developing something that can infer (speech-to-text + LLM) locally on the social worker's iPhone while they are out and about in the neighbourhoods. Responsiveness, relevance, and privacy are key design goals.

[awni]

We are actively working on a swift front end. I can't give a timeline yet but we like to move quicly :) and it's a top priority for us based on feedback from you all. Feel free to continue to post here if you have any comments / suggestions and watch this issue for more updates on progress.

[rovo79]

I'm trying to get up-to-speed(very unlikely that will happen), but does a "swift front end" mean the possibility of providing a pathway to execute some layers of MLX models on the Neural Engine via CoreML?

I semi-understand the technical constraints making integration between MLX and the ANE non-trivial, and arguably limiting the framework to a narrow set of layers. However, it seems like even a one-way trip from MLX -> CoreML would be beneficial for testing and performance. MLX's unified memory architecture for efficient training, but route eligible sub-computations to the ANE at inference time for optimal performance.

...of course these are early days for MLX!

Apologies for my fuzzy logic, I've no doubt there is a great deal I'm not taking into account.

[vade]

@rovo79 MLX as I understand it wont leverage the ANE, see this closed issue https://github.com/ml-explore/mlx/issues/18

That said, CoreML allows you to expose 'custom layers' which could be implemented in MLX, and since MLX leverages unified memory, if you were careful it would be zero copy between GPU or CPU and ANE for layer memory which is a huge win.

This is non trivial but doable. For info on Custom CoreML Layers check out @hollance's amazing CoreML Resources, specifically: https://github.com/hollance/CoreML-Custom-Layers

[dougdew64]

@awni does your statement about active work on a swift front end mean that there's an effort happening somewhere that is not visible in this repo but will eventually be merged into this repo?

[vmanot]

@awni I'm eager and happy to personally contribute to any efforts towards a Swift frontend, same question as @dougdew64 (irregardless of timeline).

[awni]

I'm eager and happy to personally contribute to any efforts towards a Swift frontend

That's awesome and encouraged!

does your statement about active work on a swift front end mean that there's an effort happening somewhere that is not visible in this repo but will eventually be merged into this repo

Yes, we are working on it. There will definitely still be lot's of to contribute, we just want to get the scaffolding right / bridging between APIs done correctly before we merge it.

[dougdew64]

@awni Although I'm grateful that there will be a swift front end, I hope that performance of the c++ code will not be significantly compromised by the introduction of that swift front end. If the swift front end has to be built atop only the highest level of the c++ api in order to maintain high c++ performance, so be it.

While profiling llama2.c and llama2.cpp I was reminded how performance can be negatively impacted by making code friendlier. In my opinion, llama2.cpp's code is much friendlier (e.g. self-documenting) than llama2.c's code. Yet, llama2.cpp seems to run much more slowly than llama2.c (~12 tokens / sec versus ~19 tokens / sec during my test). So, although I might prefer to learn how LLMs work by reading and debugging the llama2.cpp code, I'd prefer to use the llama2.c code.

Also, I hope that providing c++ equivalents of the already-existing python stuff such as the NN layers will receive higher priority than providing the swift front end.

Basically, at the top of my wish list is for this repo (or the examples repo) to contain c++-based demonstrations of running llama2 models and mistral MoE models.

[awni]

I hope that performance of the c++ code will not be significantly compromised by the introduction of that swift front end.

The C++ API will not be affected by the swift front end!

[awni]

Basically, at the top of my wish list is for this repo (or the examples repo) to contain c++-based demonstrations of running llama2 models and mistral MoE models.

@dougdew64 I'm curious do you want C++ examples or Swift examples? Why one vs the other?

Our intention with the Swift front-end is that the overhead should be minimal. So I don't expect much performance to be gained by going to C++ directly.

[dougdew64]

@awni if I could have only one, I'd choose the Swift examples.

The existence of Swift examples would imply that Swift <--> C++ interop had been implemented. And having that interop would mean being able to use MLX with Apple platform features such as SwiftUI. And that would be useful. For example, for my personal project of implementing a SwiftUI-based LLM visualization app, which I intend to be kind of like https://bbycroft.net/llm.

That said, if Apple's platform features were more available to C++, then I would not be interested in Swift interop for MLX and therefore not interested in Swift examples. I would strongly prefer to write math/physics/ML code using C++ instead of Swift, and would be willing to implement UI using C++, just as the Qt and WinUI folks are able to do.

I'm grateful that you asked for my opinion. But probably, you should discount my request for C++ examples as I am very likely an outlier. ;-)

[dougdew64]

@awni for what it's worth, while awaiting the Swift <--> C++ interop stuff for MLX, I'm attempting to re-implement llama2.cpp atop MLX to better familiarize myself with MLX. It's likely that by the time that I complete that exercise I won't need MLX C++ examples anymore.

And, some of the C++ tests serve as useful examples. For example, I'm referencing array_tests.cpp at the moment.

[dougdew64]

@awni here is an instance in which a C++ example would be helpful: https://github.com/ml-explore/mlx/issues/214

On the one hand, it's educational to have things go wrong. On the other hand, I'd kind of like to at least get llama2.cpp running atop MLX before experiencing so much education. ;-)

[dougdew64]

@awni there's an obvious (I think) question which should be asked: what will the advantage of a Swift MLX API be compared to the already-existing MPS Graph API stack? ( I left out CoreML intentionally ).

For me, the advantage is that MLX is open source and will hopefully be supported by an enthusiastic community. This stuff is interesting to me, and I'm looking forward to conversations here. But, I wonder what you and others are thinking.

[awni]

A lot of the same rational as #12 apply there as well.

[sebeaumont]

Let me add my vote to providing Swift bindings; this would be a great enabler for a product we currently in have prototype/PoC where we are aiming to do online learning of a generative model in the musical domain. The MLX project is really exciting accelerator (pun intended) for exploring just how far we can push individualised ML on desktop grade machines.

[sebeaumont]

Imagine if we got a nice MLX Swift API and Differentiable (_Differentiation) implementations!

[lin72h]

Imagine if we got a nice MLX Swift API and Differentiable (_Differentiation) implementations!

Wow, it's a powerful combination

[iliasaz]

Looking forward to Swift binding or better yet MLX Swift with full support of Differentiable! So many Swift developers (both in Apple ecosystem and Server-Side swift) would benefit from having this!

[sholtomaud]

Yeah, I mean what is mlx for? Unless there is a knockdown reason for isolating ML/AI from Swift, it's hard to understand why mlx exists, why wouldn't I just use the standard PC/NVIDIA hardware for Linux server with python/C++ api that the Swift app calls?

Bindings are a basic requirement for writing local ML iOS/macOS apps, no?

[bofeizhu]

https://forums.swift.org/t/updated-plan-for-supporting-c-interoperability-in-swift-package-manager-in-the-swift-5-9-release/65203

[AhmedHamadto]

@awni Are there plans to have mlx provide access to the apple neural engine for inference? I am trying to set up an edge computing solution, and the form factor of the Mac Mini is quite suitable for this, however, I am trying to compare the inference speed and am relatively new to most of this.

My model is built using python and anomalib patchcore. I am trying to port it over to CoreML but I've learnt that only Apple approved models utilise the neural engine.

Any guidance would be greatly appreciated.

[awni]

Are there plans to have mlx provide access to the apple neural engine for inference?

Unlikely for the foreseeable future. See discussion in #18

[mzbac]

With all the advancements of LLM, I feel all the apps will need to be modernized in order to adopt the new technology. Hence, the old way of complex UI/UX is no longer relevant if we can have on-device LLM to intercept user intentions and act accordingly (e.g., Rabbit R1). if we could provide a swift binding will definitely help accelerate that on Apple devices.

[rovo79]

@mzbac I wonder if efforts like Apple Ferret play a role in that? https://github.com/apple/ml-ferret

[awni]

Swift bindings are a work in progress (can't give a precise timeline, but it's under active development)!

[dougdew64]

@awni are C++ improvements such as slicing support and NN layers being added so that the Swift bindings may be built atop those C++ improvements? Or are the Swift bindings being built atop the already-existing C++ API?

Will the Swift bindings offer support for array slicing?

[rvsrvs]

As interesting to me would be whether or not the swift representation of the underlying layers makes of swift's move-only types (aka ~Copyable) wrapped around the underlying c++ types.

[allenschmaltz]

MLX is great! This comment requires no reply, but we just wanted to add a vote of very strong interest in the Swift bindings (for use in a macOS sandboxed app). Once they are available, we will integrate genAI model support (for uncertainty-aware generation!) directly into the Reexpress macOS app.

(Btw, also happy to do beta testing of the Swift bindings.)

[davidkoski]

As interesting to me would be whether or not the swift representation of the underlying layers makes of swift's move-only types (aka ~Copyable) wrapped around the underlying c++ types.

One difficulty in this is that ~Copyable applies to structs and structs don't have a deinit() -- this makes it hard to control the lifetime of wrapped objects.

I do think mx.array would be interesting as a struct. Aside from mutation via indexing (e.g. a[1] = 3, which really replaces the underlying storage) these behave very much like structs.

But the lifecycle requirements force it toward classes.