oruebel
commented
1 year ago Approach and Plan
For DataChunkIterators the easiest place to start I believe is to look at how to parallelize the ZarrIODataChunkIteratorQueue.exhaust_queue function.
The reason this is a good place to start is because: 1) it keeps things simple, because at that point all the setup for the file and datasets is already done and all we deal with is writing the datasets itself 2) it's a pretty simple function, and 3) it allows us to parallelize across both chunks within a dataset as well as across all DataChunkIterator that are being used. I believe this should actually cover all cases that involve writing of DataChunkIterator, because I believe the ZarrIO.write_dataset method also just calls the exhaust_queue function to write from DataChunkIterator, here:
only in this case the queue will always just contain a single DataChunkIterator object. I.e., when calling ZarrIO.write with exhaust_dci=False then we would get all DataChunkIterator objects at once as part of the queue and if exhaust_dci=True then we would see them one-at-a-time. I.e. parallelizing exhaust_queue I believe should take care of all cases related to parallel write of DataChunkIterator objects.
If we also want to do parallel write for other datasets that are being specified via numpy arrays or lists, then that would require us to also look at ZarrIO.write, but I think as a start focusing on DataChunkIterator should be fine. Also, for the more general case we also likely need to worry more about how to specifiy how to actually determine intelligently how to write these in parallel, whereas with DataChunkIterator and ZarrDataIO there is a more explicit way for the user to control things.
Identify the best injection point for parallelization parameters in the io.write() stack of HDMF-Zarr
If it needs to be parametrized on a per-dataset level then the definition of parameters would probably go into ZarrDataIO. If this needs to parametrized across datasets (i.e., use the same settings for all) then this would probably either go as parameters of ZarrIO.write or ZarrIO itself. I think this all depends on how flexible we need it to be. ZarrIO may end up having to pass these to ZarrDataChunkIteratorQueue, but that is an internal detail that a user would normally not see, since the queue is an internal data structure that is not exposed to the user (i.e., it is only stored in a private member variable and should not be used directly by the user).
CodyCBakerPhD
What would you like to see added to HDMF-ZARR?
Parallel Write Support for HDMF-Zarr
Allow NWB files written using the Zarr backend to leverage multiple threads or CPUs to enhance speed of operation. Objectives
Zarr is built to support efficient Python parallelization strategies, both multi-processing and multi-threaded HDMF-Zarr currently handles all write operations (including buffering and slicing) without exposing the necessary controls to enable these strategies
Approach and Plan Identify the best injection point for parallelization parameters in the io.write() stack of HDMF-Zarr Progress and Next Steps TODO Background and References
Is your feature request related to a problem?
No response
What solution would you like?
Identify the best injection point for parallelization parameters in the
io.write()stack of HDMF-Zarr (as controlled via theNWBZarrIO)Essentially revolving around the line https://github.com/hdmf-dev/hdmf/blob/2f9ec567ebe1df9fccb05f139d2f669661e50018/src/hdmf/backends/hdf5/h5_utils.py#L61 from the main repo (which might be what is used to delegate the command here as well?
Do you have any interest in helping implement the feature?
Yes.
Code of Conduct