Best way to implement user-defined FFI?

[VictorTaelin]

I've pushed a PR for the initial IO (#336). It works as a simple monadic FFI interface between the interaction net computations and native function calls. The core is functionality is done C, but it is still missing on CUDA. After that, the main challenge becomes: what is the best way to let the user implement their own IO?

One way to do it would be to let the user add C strings in the middle of a HVM (and Bend) file. For example, a custom put_text could be implemented as:

// foo.hvm

#FFI C {{
  Port io_put_text(Net* net, Book* book, Port argm) {
    // Converts argument to C string
    Str str = read_str(net, book, argm);
    // Prints it
    printf("%s", str.text_buf);
    // Returns result (in this case, just an eraser)
    return new_port(ERA, 0);
  }
}}

@main = ...

This would then be treated in two ways:

1. For the interpreted version, we would invoke a C compiler to generate a dynamic lib, load it on the program, and push it to book->ffns_buf.

2. For the compiled version, we would just add it to the book as if they were builtin ffns (see the book_init fn).

While this works, this sounds a little bit hacky, since the user would have to use internal functions like read_str and node_load, so, we'd end up coupling the HVM file format to lots of internal functions name and conventions, preventing us from refactoring later. An alternative would be to pre-readback the Port into a C structure (converting λ-encodings to C numbers, strings, trees, etc.). But that introduces complexity and inefficiency. For example, a real-time image renderer probably wants to use a custom, parallel read_img function which operates directly on the inet memory. And other than these concerns, I'm also not sure including C strings like that from the middle of .hvm files is really an elegant approach, but I'm not sure what else we could possibly do.

[developedby]

Why not have the user specify C files to include and the names of functions that should be registered? That would be more flexible and portable (someone could for example have the same HVM program but swap the C implementation of the foreign functions)

[developedby]

Including C source code will not scale well and will fail for anything slightly more complex.

What I think most languages do is have the ability to load arbitrary compiled object files (libraries, dlls, etc) and a builtin way of registering functions in that loaded library.

Even if dinamically loading libraries is too much (would require a somewhat portable way of calling the system's loader), for statically including libraries that would be enough. We'd just need some directive that specifies the compiled library and the C header that explains it.

[VictorTaelin]

How do we statically link libraries for the run-c command though, given that it is an interpreter which is using the already pre-compiled hvm.c file? As for dynamically loading, perhaps that could work...

[enricozb]

We did dylibs in hvm-core and it seemed to work well. Had to do some version checks / warnings though.

[developedby]

I meant that for statically including FFI, we can rely on the compiler/linker, while dinamically loading is a bit more complicated and harder to support for different platforms.

[enricozb]

As IO is being finalized, my initial thoughts of the design exposed to users would be the following:

• add two IO functions DL_LOAD(file String) -> Opaque and DL_CALL(fn_name String, Port arg) -> Port
• use dlopen to load dynamic libraries and call into their symbols
• FFI functions would have the same signature as IO functions (on the C runtime Port (*func)(Net*, Book*, Port)).

There are some remaining implementation questions though:

• With this is that user's can't do much with just a Port without having access to some utility functions (like converting from/to lists), but extracting this functionality is non-trivial.
• Actions that depends on the allocator (like creating and returning list) is more complex to factor out.

[enricozb]

Initial attempt here: https://github.com/HigherOrderCO/HVM/pull/394