search

earth-mover / icechunk

Open-source, cloud-native transactional tensor storage engine
https://icechunk.io
Apache License 2.0
261 stars 14 forks source link

Add a C-compatible API #381

Open asinghvi17 opened 5 hours ago

asinghvi17 commented 5 hours ago

It would be great to have a C-accessible API for Icechunk, as a "lowest common denominator" for access from Julia, C, GDAL, etc.

I am personally a contributor to Julia's Zarr implementation in https://github.com/JuliaIO/Zarr.jl, and would be happy to write a Julia integration for Icechunk given a C API from this end. Interop from Julia to Rust directly is tricky (it's a similar problem as C++ has) so a C-API is the easiest way to go here.

I imagine it could be structured pretty simply, the store would likely have to be represented as a pointer. C doesn't look to have many async capabilities but that can be handled on the calling language's end (Julia has async calls, for example, that live in a specific pool of threads).

Would be great to hear what might be needed to support this!

cc @alex-s-gardner @felixcremer @meggart from the Julia end

rabernat commented 5 hours ago

Yes, we think this would be a great addition and support it 💯 percent!

It would be great to find a contributor who has experience creating C bindings from Rust libraries to work on this. @asinghvi17 is that something you have any experience with?