Open kyoungrok0517 opened 1 year ago
"Append" basically corresponds to resizing the bounds of one dimension, and then writing to the new portion.
You can do these steps separately in TensorStore currently (resize
and then write
) but that will not work correctly if done concurrently from multiple machines. Note that zarr-python does not have any special atomic append support, it is just a convenience interface for resizing and then writing.
There are a few issues with making this work correctly:
Is there a resize function for the zarr3 driver in C++? How does that work? Can I change the dimensions with it? I have a 1D {N^6}array that needs to be reshaped to {N^3,N^3} and {N^2,N^2,N^2}.
Hello. First I'd like to refer to my previous issue #67, which explains my use case.
I want to append embeddings to tensorstore from multiple processes of pods. There is
append()
inzarr
, but I couldn't find a equivalent function in tensorstore. How can I achieve the similar in tensorstore? below is my writing code. Thanks!