search

nathanin / micron2

0 stars 0 forks source link

Parallel write to hdf5 #33

Closed nathanin closed 3 years ago

nathanin commented 3 years ago

Reasoning: dataset creation is per-channel, and technically can be parallelized. We can write a single dataset for each channel and merge later, or create several in parallel within the same hdf5 file. Which is better: a pipelined solution, or an at-once solution? Does parallel writing require non-standard packages?

Possible? https://stackoverflow.com/a/51708858

nathanin commented 3 years ago

Closing. Planning to abandon the hdf5 dataset and related workflows in future updates. Current functionality is good as-is until the replacement workflow is implemented