$250 - Common HL/HXCPP native C API binding format

[larsiusprime]

Introductory Information:

Hashlink is basically the successor Neko in terms of Haxe VM targets, with the added advantage of having both a compiled-C and a bytecode target.

HXCPP is the traditional workhorse of Haxe native C++ targets.

Hashlink promises some advantages to workflow -- particularly faster compilation speed during development, but it's still relatively new and most of the C/C++ bindings for existing libraries for Haxe native targets are still written for HXCPP, and HXCPP is a solid well-maintained target that will surely see continued use alongside HashLink. This puts native extension library maintainers in a dilemma, do we support HXCPP or Hashlink, or both? Right now supporting both means maintaining two parallel sets of bindings, which is not only double the work, but also fertile ground for subtle duplication/oops-I-forgot-to-update-the-other-one errors.

This bounty is for developing a common format for writing native extensions to C API's for Haxe, that is compatible with both HXCPP and HashLink.

This is a separate issue from automatically generating native extensions with minimal effort from, say, raw C files. It's okay for the sake of this bounty if the accepted solution requires a bit of manual labor in writing the bindings themselves, so long as each binding doesn't have to be written twice, once for HXCPP and once for HashLink.

The Project

• Produce a method that bridges a C API to Haxe
• The solution should produce both HL bindings and HXCPP bindings
• The programmer should only have to write an "exposure" for a given C API function once, and get both bindings out of it
• Provide a working proof of concept example of limited scope
• Open source the whole shebang under MIT license or equivalent
• The exposed Haxe interface should match the C interface as closely as possible
• Absolute minimum dependencies
• Account for and note any "leaky abstractions" / incompatibilities between HXCPP & HL
• Minimize use of operations that generate expensive side-effects (such as hidden Dynamics on HXCPP target) -- libraries that rely on this solution should be suitable for high-performance things like physics and rendering.
• Exact approach (e.g. using CFFI, CFFI Prime, C externs, untyped C blocks, etc) is entirely up to you, just provide a reasoning for why your solution is the best one you could think of compared to the alternatives.

The sample should:

• Demonstrate passing two numbers from Haxe, adding them together on the C side, and returning them to Haxe
• Demonstrate passing a file name from Haxe, loading the file and resolving it as raw bytes on the C side, and returning the bytes to Haxe
• Demonstrate passing two (ASCII) strings from Haxe, concatenate them together on the C side, and return them to Haxe
• Demonstrate all the above, with the functions implemented in loose .c and/or .h files
• Demonstrate all the above, with the functions implemented in a .dll file
• Be able to be compiled and run against hxcpp target and hl target with a simple .hxml script

Out of scope

None of these things are part of the project:

• C++ API support (we only care about supporting plain C API's for now, save this for later)
• Exhaustive documentation (just a simple example and a quick description is fine)
• Unicode string support in the sample (just ASCII is fine for now)
• Pass a structured object from Haxe, mutate some of its values on the C side, and return it to Haxe
• Properly handle a C api that asks for a C data structure as input (important to solve eventually but let's not worry about it just yet)
• Automatic generation of bindings from a minimal description (such as c header files alone or whatever)

Prerequisites

• Knowledge of HXCPP
• Knowledge of Hashlink
• Knowledge of Macros
• Knowledge of C

What's been done so far

Nothing specifically on this project, but here's some prior work to learn form:

• Linc -- solely focused on HXCPP
• Everything I know about CFFI / CFFI Prime
• Haxe externs
• Hashlink links:
  • https://github.com/HaxeFoundation/hashlink/wiki/C-API-Documentation
  • https://haxe.org/blog/hashlink-indepth/
• Updates from comments:
  • Nadako's JNI stuff
  • Nicolas' WebIDL (generate emscriptem/wasm+HL from IDL files)

How Much can Lars help?

• I'm available to talk to and ask/answer questions but you'll be doing the majority if not all the work here. I recognize I'm out of my depth.

What remains to be done

• The whole thing!

Budget/Timeline:

• No specific timeline. 1 month would be great? I'd like to see it done before November.

Budget is $100 USD. $200 USD $250 USD

[nadako]

For the people looking into this:

I experimented with auto-generating glue for Java's JNI API for hashlink before and had some success. The idea is pretty simple: we define an extern class in Haxe like this and then process it with a macro and generate glue C code somewhat like this. That particular code is purely experimental and JNI-specific of course, but I think the idea is really nice and the same approach can be used for generating both HL and HXCPP glue code.

[ncannasse]

I have also some experiment to generate both Emscriptem and HL bindings from an IDL structure, see https://github.com/ncannasse/webidl

[larsiusprime]

@ncannasse -- very interesting, Nicolas! Do you actually use it in production anywhere or is it just tinkering for now? Presumably this format could be used to generate hxcpp bindings in addition to your emscriptem/HL bindings?

[ncannasse]

@larsiusprime not yet in production, but I did use it for some Bullet3D experiments here https://github.com/HaxeFoundation/hashlink/tree/master/libs/bullet

The nice thing it that it generates both Haxe externs (using macros) and C stub code from IDL. Cons is that it only supports C++ for now, but I guess the IDL could be enriched with additional annotations to support C.

One big cons versus "manual" wrapper is that it does not allow much cooperation with GC so it requires for the Haxe wrapper objects to keep references on the C++ objects. And when the the Haxe object gets GC'ed to delete the C++ object automatically. This also requires to make sure that Haxe references are keeping each other the way C++ objects are, which is extra post wrapping work.

[larsiusprime]

@ncannasse speaking of "manual" wrapper -- is it possible in principle to write the "manual" part, with GC fiddling and all, and have a script/macro/etc generate both a HXCPP and a Hashlink binding from whatever intermediate format is used? Or is the way GC behaves in Hashlink and HXCPP sufficiently different to make this not possible?

[kevinresol]

Any ways to pool a budget? Maybe something like OpenBounty?

[ncannasse]

I think the problem of "automatic" wrapper generation VS an IDL and the problem of a "common" way to express the wrappers are two separate problems. Once the "common" problem is solved, it will help a lot the "automatic" problem.

Regarding a good way to express the "common" wrapper generation, there are two ways to deal with it:

a) write C code with an #include <common_wrapper.h> that defines an API that behind the scene will call HL API or HxCPP one (or other platforms). But this is quite low level and I'm not sure how much additional platform-specific things will be required there.

b) write Haxe code that also implements a cross platform "extern" API, that does not get compiled but that gets read by macros to output the corresponding platform-specific C code. That's slightly higher level and allows to add platform specific constructors automatically by detecting the places they are needed from the Haxe AST.

[larsiusprime]

@kevinresol -- possibly! So far I've just done everything informally. Rest assured, if several people contribute something significant I'll do my best to split the bounty fairly, and maybe even bump it up a little if that feels necessary.

[kevinresol]

I mean the other way round, I probably would like to contribute some money to the budget

[larsiusprime]

@kevinresol -- oh, interesting! That's a swell idea, can you link to any services you like? What's this OpenBounty thing for instance?

[kevinresol]

Not really, I just randomly searched it without deeply looking at it yet. Maybe we should discuss in a new thread.

[larsiusprime]

@kevinresol: Great, let's discuss that here.

@ncannasse: Thanks for the input! Re this:

I think the problem of "automatic" wrapper generation VS an IDL and the problem of a "common" way to express the wrappers are two separate problems. Once the "common" problem is solved, it will help a lot the "automatic" problem.

I'll update the issue to indicate the scope of this bounty is solely the "common" problem. If we can solve that, we can tackle the "automatic" problem as a separate issue.

[kevinresol]

How about extracting (one of) the libs into a separate repo to demonstrate how to write a native lib for hl?

Take mysql as an example, move all the mysql source, glue code, haxe extern files, build instructions (cmake) into a repo which can be consumed by simply adding sth like -lib hl-mysql to a hl project. This is similar to what linc does. I actually like linc because I found its self-contained approach makes it much easier to develop and consume native extensions.

[kevinresol]

How about extracting (one of) the libs into a separate repo to demonstrate how to write a native lib for hl?

Take mysql as an example, move all the mysql source, glue code, haxe extern files, build instructions (cmake) into a repo which can be consumed by simply adding sth like -lib hl-mysql to a hl project. This is similar to what linc does. I actually like linc because I found its self-contained approach makes it much easier to develop and consume native extensions.

@ncannasse what's your opinion on this?

[ncannasse]

@kevinresol on one hand I like the idea , otoh this would require users of these libraries to install Visual Studio C++ in order to compile them. I kind of like that you can ship precompiled binaries so users don't have to deal with native compilation at all -- that works especially for windows users.

[kevinresol]

@ncannasse Ideally haxeilb should be able to selectively download files per user's current OS. npm does that smoothly and is even able to compile the binaries during a library install.

Anyway, compared to the binary delivery problem I think I am far more concerned about the fact that the current approach with hl extensions is not scalable at all. To be honest, I am unaware of any HL extensions outside the HL repo.

In the early days hxcpp suffered from the problem as well. At that time native extensions are mostly tied to OpenFL (because OpenFL provided a native binding mechanism? Not very sure). But since the introduction of linc, there are so many framework-agnostic libraries for hxcpp emerged which is really helped users a lot.

[larsiusprime]

Yeah, I'm a big fan of the linc standard!

FYI, I'm bumping the value of this bounty to $200.

[larsiusprime]

Bumping the value of this bounty to $250

[Sunjammer]

linc + inline cpp has been a revelation, but it's less a standard than a guideline. From lib to lib Linc authors still pick divergent approaches, especially when it comes to managing call backs. I don't necessarily mind this, it reflects the different designs in the c libraries themselves. Hard to normalize this.

[chfoo]

I had some time looking at this and I created a proof-of-concept. It basically is a command line tool that reads in XML files and generates a Haxe extern class file and a C function wrapper file. The result is a one-to-one mapping of parameters but only of very limited types for now.

I used externs because it was the approach used in the standard library which provided lots of examples. The process is a bit fragile as it simply spits out Haxe and C code without any framework or error checking. The example might show warnings or not compile, particularly in HashLink, because I am using uint64_t* to bind to size_t* for the sake of simplicity. But overall, it can pass numbers, raw bytes, and strings (with caveats noted in the readme).

[Aurel300]

Let me introduce ammer.

class Adder extends Library<"adder"> {
  public static function add_numbers(a:Int, b:Int):Int;
  public static function load_file(filename:String, loaded:SizeOfReturn):Bytes;
  public static function concat_strings(a:String, b:String):String;
  public static function reverse_bytes(data:Bytes, dataLen:SizeOf<"data">):SameSizeAs<Bytes, "data">;
}

Add some hxml defines and that's it!

Some notes re: the bounty requirements:

The exposed Haxe interface should match the C interface as closely as possible

Account for and note any "leaky abstractions" / incompatibilities between HXCPP & HL

The interface is matching except where dealing with e.g. pointers. See the Bytes section. Haxe does not support direct memory manipulation, so the library must map a Bytes object to a pointer and size.

Demonstrate all the above, with the functions implemented in loose .c and/or .h files

Compiling the actual library from C sources into a dynamic library (dll, dylib, so) is not part of the project, and should be done with existing tools - gcc/clang, make. ammer assumes you have a dynamic library and its headers.

[larsiusprime]

Whoa! I'd almost forgotten about this. Ping me on Monday so I don't forget to review this!

[Aurel300]

@larsiusprime ping :)

[larsiusprime]

Cool, @Aurel300 -- so catch me up a bit here. You're claiming this is done and fulfills all the bounty requirements? (I'll be happy to settle for "Most" depending on circumstances, too).

Got a easy to use sample project I can compile and play with right away?

[Aurel300]

All of the requirements of the OP are fulfilled except as noted in the previous post. Do note that the minimal example you ask for requires very few types, and as a result, I haven't implemented much more :) See the types table, as well as the types issue which lists some things that I will be implementing soon.

The sample project is in the repo. The setup will be basically:

$ haxelib dev ammer <path to clone>
$ cd ammer/samples/poc/native
$ <compile the native library somehow, preferably place it in /usr/local/lib or equivalent>
$ cd ..
$ haxe build-hl.hxml
$ hl bin/hl/sample.hl

(I'll be on Discord if you run into any problems!)

[larsiusprime]

Great, this looks pretty dang solid so I'm not too worried! I'll test it now, what's your discord handle?

[Aurel300]

I messaged you already but perhaps you changed logins? Aurel300#1177

[larsiusprime]

Ah, no I just don't have discord open!

[larsiusprime]

This bounty has been claimed! I will shortly pay the full amount out to @Aurel300. Well done!