Open constantinpape opened 6 years ago
Thanks for opening this.
My naive thinking when suggesting we add C bindings is we coerce C pointers into something C++/xtensor can handle. Perhaps xt::adapt
would work? Really whatever xtensor does for NumPy arrays should be pretty insightful.
In terms of handling C arrays, xnd (particularly the C portion as there is a Python portion as well) may be valuable. It is very young and under active development. Not aware of many other N-D options in C. Based on my brief search came across a few C implementations like this that were for 1-D arrays only and generally some combination of the following: unmaintained, GPL'd, or part of a much larger library.
Handling type parameterization in C basically is done one of a few ways (may be more that I'm missing):
void*
(or char*
) with length#define
-style templates__Generic
ref: https://softwareengineering.stackexchange.com/q/262571 ref: https://www.gnu.org/software/autogen/ ref: https://github.com/numpy/numpy/blob/master/tools/npy_tempita/__init__.py ref: http://jinja.pocoo.org/docs/ ref: https://en.cppreference.com/w/c/language/generic
Yes, using xt::adapt
would probably work as well.
The solution would probably include passing a 1D C-array as well as the shape (long array + number of dimensons?) to the read/write function, get a xt::view
to it with xt::adapt
and then use xt::reshape
to get a ND-view.
Thanks for the references on parameterization, I will have a closer look soon.
We might also want to pass in an enum
for type selection. Then we would get something like this (roughly).
#include <cstddef>
#include <numeric>
#include <functional>
#include <vector>
#include "xtensor/xadapt.hpp"
enum DataType {
...
INT,
...
};
void do_something(void* data, enum DataType type, size_t ndim, const size_t* shape)
{
std::vector<std::size_t> shape_vec(shape, shape + ndim);
std::size_t size = std::accumulate(
shape_vec.begin(),
shape_vec.end(),
std::size_t(1),
std::multiplies<std::size_t>()
);
switch (type) {
...
case INT:
{
auto a = xt::adapt((int*)data, size, xt::no_ownership(), shape_vec);
}
break;
...
}
}
Also raised issues with xnd
( https://github.com/plures/xnd/issues/31 ) and xtensor
( https://github.com/QuantStack/xtensor/issues/1030 ) about being able to interoperate between the two libraries, which would provide another way to go between C and C++. Given they both seem to receive funding from QuantStack, expect this work would be prioritized by both projects.
Edit: Strikethrough due to an inaccuracy.
xtensor is funded by QuantStack but xnd is funded by Quansight.
Thanks @davidbrochart. Must have misread. Have edited the comment above accordingly.
Since you are here, @davidbrochart, do you have any thoughts on how we might modify the example above to use xnd instead? It would be nice to not have to pass through all of the array info piecewise, but am not familiar enough with xnd to say how.
@jakirkham I think your example is pretty much what we will need for read/write.
However, we will also need to pass a pointer to the Dataset
.
As a side note, a C implementation would be a good in road for Fortran as well.
I'm sure it's occurred to you already, but with this in mind it would make sense to put all of the attribute access stuff through C++ as well and have python just bind to that.
Yep, would make sense. I did not write it that way when working on the python bindings, because the arguments in the glue code need to be properly typed, and I didn't want to write functions for all possible attribute values. However, it should be possible to bypass this by dumping the attributes to a json formatted string and feeding this to the glue code.
cc @wardf @DennisHeimbigner (Unidata NetCDF developers)
This looks interesting; we're currently discussing our approach to adding native zarr functionality to the core netcdf C library, and having a pre-existing library we can link against would lighten the technical burden. On the other hand, once it's in the C library it might be a simple matter to expose it for more general usage. This is an interesting discussion, thanks for the tag @rsignell-usgs
Cool, glad to hear this would be useful for NetCDF. 😄
Do you have any thoughts on or suggestions for the proposal thus far, @WardF?
Let me take a closer look and follow up with thoughts @jakirkham :)
Not sure if our work is helpful. We implemented c wrapper for z5 and it is just an experimental project. see here: https://github.com/kmpaul/cz5test
Feel free to let me know if any improvement is needed.
@weilewei I had a very quick look and it looks promising. I don't have much time this week, I will try to have a closer look in the next week.
Also, to give some context on the larger ecosystem here, there's currently discussion for a new version of the zarr spec going on, that will at some point unify the zarr and n5 formats, see issues and PRs in https://github.com/zarr-developers/zarr-specs. Once the spec is there, the plan is to provide a C implementation of this spec, that would then also be used by z5 internally for the basic operations (I/O of chunks, compression).
Unfortunately, as far as I know, there is not a clear timeline for this yet and having a c wrapper for z5 will be useful in any case, just wanted to inform you that potentially there will some other things happen in this domain.
@constantinpape thanks for pointing me to the zarr specs discussion. Right, if you have a pure c implementation following the zarr spec than we do not need c api wrapper.
The C wrapper I wrote is just for my own experimental project. Its interface is currently not very perfect but most of file io functionalities are there and tested. I will improve it later. Any suggestion is welcome.
@weilewei @halehawk I forgot about your efforts in writing a C wrapper and was just reminded of this. Just wanted to say that this is a very valuable contribution, especially because the timeline for the native zarr c implementation is still unclear. I would like to link to your effort in the README here. Is https://github.com/kmpaul/cz5test stil the relevant repository for this?
@constantinpape The link is correct. Thanks for bringing it up and linking our repos!
The possibility in wrapping the z5 API for C came up in https://github.com/zarr-developers/zarr/issues/285#issuecomment-409438299 .
As far as I can tell, this would involve two main tasks: Exposing the C-API of
Dataset
/DatasetTyped
: https://github.com/constantinpape/z5/blob/master/include/z5/dataset.hxx#L25And implementing
read/writeSubarray
for C-arrays, similar to https://github.com/constantinpape/z5/blob/master/include/z5/multiarray/xtensor_access.hxxThe first task should be straightforward. Though I am not sure what's the best approach for wrapping all datatypes without templates. I hope there would be a more clever way than redundant wrapper code for each individual datatype.
The second task will probably involve some pointer arithmetic, but maybe some code I have written for fast copy in xtensor could be re-used: https://github.com/constantinpape/z5/blob/master/include/z5/multiarray/xtensor_util.hxx#L45