[RFC] Future directions for cbindgen

[eqrion]

I've wrote a blog post talking about some old persistent issues in cbindgen coming from the original architecture [1], and a potential forward path [2].

This would solve:

1. 203

2. 7

3. 139

4. 183

And probably others. Comments and feedback are welcome!

[1] http://dreamingofbits.com/post/future-directions-for-cbindgen-rust-ffi/ [2] https://github.com/eqrion/rust-ffi

[konstin]

I just want to add the two use case I see: milksnake is currently build.rs integration to generate the headers. It's possible to make cbindgen a python package itself (e.g. by just putting the binary into a python package) though. For pyo3-pack I'd like to include cbindgen directly so that users don't need any extra tools, so having something like the current "give me a crate path and some options and I'll write you header" api would be nice.

I also wonder whether the rls wood be a good fit here as a stable abstraction over the compiler. This would afaik also avoid pulling rustc in.

[eqrion]

I just want to add the two use case I see: milksnake is currently build.rs integration to generate the headers. It's possible to make cbindgen a python package itself (e.g. by just putting the binary into a python package) though. For pyo3-pack I'd like to include cbindgen directly so that users don't need any extra tools, so having something like the current "give me a crate path and some options and I'll write you header" api would be nice.

I also wonder whether the rls wood be a good fit here as a stable abstraction over the compiler. This would afaik also avoid pulling rustc in.

I looked into rls and save-analysis a bit [1], and unfortunately I don't think it has enough information for generating correct headers. I'd also be concerned about relying on a tool designed for IDE's here, as it seems like we have different requirements.

I've also been pointed to rust-analyser [2] as another potential solution, but I haven't been able look into it much. It might have similar issues.

[1] https://github.com/nrc/rls-rustc [2] https://github.com/matklad/rust-analyzer

[Moxinilian]

I've read your blog article and I am very interested by the rust-ffi idea. I am a core member of the Amethyst game engine organization, and in our initiative to offer built-in scripting support in the engine, we designed a general-purpose interface for any language to bind to the engine (in a manner that is specific to Amethyst). Part of this design includes exposing some of the Rust functions and types to the scripting languages through FFI, and thanks to the awesome capabilities of cbindgen this can be done very easily. This is a good start, but very limited: how can we express complex patterns such as iterators or even more simply structure methods to the scripting languages so that they can generate idiomatic type interfaces somewhat automatically?

This is why I'm suggesting that the format for ffi.json contains more than just data to build C-like FFI. If it contained additional information, such as for example trait implements, a custom converter could make the bindings automatically more idiomatic, depending on the receiving programming language.

For example, if something implements Iterator, then this knowledge can be used by a Lua binding generator to add the specific flags to the type so that it can be used directly in a Lua for loop.

I suggested adding trait data, but I'm not sure this would be the best thing to have the most general-purpose thing possible. It has the merit of being relatively easy to implement, but it might also miss some other information on the code (for example methods associated with structs).

What do you think of this idea?

[eqrion]

@Moxinilian One of my original goals with cbindgen was to eventually extend it to generate idiomatic C/C++ code for idiomatic Rust code. Roughly supporting rust things like iterators, drop, rc, box, inherent impl methods with idiomatic C/C++ wrappers.

So I'd love for this to be done (even better for more languages than just C/C++), but it's been more or less dropped in the focus to just get FFI semantics correct.

I think we could do a couple things with a rust-ffi like solution to make this a reality.

1. Serialize all attributes on items to ffi.json
2. Write a proc-macro crate for generating appropriate rust extern "C" glue code for common rust idioms when there's a marker attribute (e.g. #[rust_ffi_derive])
3. Extend the binding generators for each target language to look for #[rust_ffi_derive] and generate language specific code to bind to the extern "C" glue code

What I like about this approach is that it moves the problem from the rustc plugin to proc-macros and binding generators. Otherwise I'd be concerned about the churn in the plugin and the ffi.json format.

[konstin]

I just want to add rust-analyzer here because it could solve the problems stated in the blog post without requiring rustc itself.

[Gankra]

I discussed with the rust-analyzer folks and while they would love to be a viable solution for cbindgen's issues in the long-term, they just won't be ready to do so for a while.