Are write operations with the zarr Driver guaranteed to be thread- and process-safe?

[xantho09]

Suppose I have an existing on-disk Zarr array. If I were to have two separate processes that:

1. Open this Zarr array via tensorstore.open
2. Write to separate regions that potentially share the same chunks within the Zarr array

Are these two write operations guaranteed to write correctly?

For example, suppose my.zarr has a chunk shape of (64,64,64).

• Process 1 writes to (0,0,0):(64,64,32)
• Process 2 writes to (0,0,32):(64,64,64)

# Process 1
path = "path/to/my.zarr"
arr = ts.open(
    {
        "driver": "zarr",
        "kvstore": {"driver": "file", "path": path},
    },
    open=True,
    read=True,
    write=True,
    create=False,
).result()

arr[(0,0,0):(64,64,32)] = 100

# Process 2
path = "path/to/my.zarr"
arr = ts.open(...) # Same as Process 1

arr[(0,0,32):(64,64,64)] = 200

The only mention I could find was in the homepage, under the list of highlights.

Supports safe, efficient access from multiple processes and machines via optimistic concurrency.

And some basic testing seems to suggest that this is indeed true.

However, is this guaranteed to be the case? Is there anything within the documentation that provides this guarantee?

P.S. Out of curiosity, how is the OCC actually implemented? Checking the last modified date of the Zarr chunk in which to write, or something along these lines? P.P.S. Great library, by the way

[laramiel]

To achieve this you need to use transactions, note, however, that this will not work with the s3 driver.

[xantho09]

I see...

So something like this would be sufficient. Is that correct?

# Process 1
path = "path/to/my.zarr"
arr = ts.open(
    {
        "driver": "zarr",
        "kvstore": {"driver": "file", "path": path},
    },
    open=True,
    read=True,
    write=True,
    create=False,
).result()

with ts.Transaction() as txn:
  arr.with_transaction(txn)[(0,0,0):(64,64,32)] = 100

# Process 2
path = "path/to/my.zarr"
arr = ts.open(...) # Same as Process 1

with ts.Transaction() as txn:
    arr.with_transaction(txn)[(0,0,32):(64,64,64)] = 200

I do have some additional related questions:

• Suppose the two processes write to the regions (0,0,0):(900,900,650) and (0,0,650):(900,900,1280), where the regions are fairly large and only share a comparatively small number of common chunks. I'll refer to these two transactions as T1 and T2.

  If the transaction commits happened to clash and T2 needed to be rolled back and retried, does the entirety of T2 get rolled back and retried, or is this limited to the overlapping region of (0,0,640):(900,900,704)?

• If process-safety isn't a requirement, roughly how much slower is performing write operations via transactions versus calling TensorStore.write() directly?

  For simplicity, I'm limiting the scope to only the Zarr driver with an on-disk Zarr array.

Please and thank you.