Open hellishfire opened 2 weeks ago
Java doesn't really have a concept of slicing, so either this needs to copy the data (at least for things that can't easily be sliced, like the validity buffer) or it should error on import.
Java doesn't really have a concept of slicing, so either this needs to copy the data (at least for things that can't easily be sliced, like the validity buffer) or it should error on import.
I took a quick look at the slice() method of VectorSchemaRoot, which slices the data buffer, and slice or copy the validity buffer
Yes, but that's different in intent than the C++ version (which just tracks an offset). Here it means we can't do a 0 copy import.
Yes, but that's different in intent than the C++ version (which just tracks an offset). Here it means we can't do a 0 copy import.
I understand that, but IMO data correctness comes before performance, so a copy of validity buffer is tolerable in this case?
CC @vibhatha
We also have to traverse recursively since child arrays can have their own offset
I will try to reproduce this and evaluate the approach suggested by @lidavidm. I will probably need some time to try this out.
Describe the bug, including details regarding any error messages, version, and platform.
Reproduced on latest arrow release (16.0)
When importing a sliced RecordBatch from c to java
On c side:
On java side:
The imported vectorSchemaRoot maintains the correct length(which is 2), but the offset info (which is 8) is not respected, hence the content of the imported vectorSchemaRoot points to the first 2 rows of the original_record_batch, while the desired content is sliced_record_batch.
I'm not familiar with arrow code, but it seems that the offset info is actually present in org.apache.arrow.c.ArrowArray.Snapshot, but org.apache.arrow.c.ArrayImporter ignores the offset in org.apache.arrow.c.ArrayImporter.doImport(ArrowArray.Snapshot)
Component(s)
Java