search

earth-mover / icechunk

Open-source, cloud-native transactional tensor storage engine
https://icechunk.io
Apache License 2.0
291 stars 17 forks source link

Discussion: Creating a C Interface #399

Open meggart opened 2 days ago

meggart commented 2 days ago

This is my first attempt of a proof-of-concept Rust -> C -> Julia roundtrip that was discussed in #381 where I just wrapped the first function calls of the basic example that create a repository and adds some groups. Whoever is interested, maybe you can have a quick look and let me know your opinions. A C header file is generated, but currently the c wrapper in icechunk-julia is manually generated. Of course, if one wants to roll out a full c lib for the storage interface there would be a few design decisions to make. The most burning questions I would have are the following:

  1. To what extent would one expose rust structs through the C API as well. For example things like RepositoryConfig could be marked with [repr(C)] to get C memory layout which would make sharing them much more convenient. In the end only things wrapped in Arcs actually need to be hidden behind opaque pointers.
  2. Some system of automated error code translation would be nice, in the example here I only return a generic -1 in case of any error
  3. Also important to note that any panic on the rust side leads to a segmentation fault on the caller side, to it is crucial to carefully unwrap every result
  4. In this example I use calls to block_on to get out of the async machinery. However I have no idea how this would interact if a calling site actually tried to perform async calls into the library (probably it would just break?)