Currently, H5Spark parallelize the IO along the slowest dimension, i.e., the dimension that changes slowest on disks. For example, for a 2D C array x[10][200], the h5spark will choose the first dimension to partition, and then the maximum partition it can have is only 10, which is also the maximum degree of parallelism.
If we want to parallelize along any user-specified dimension, the current code needs a little bit modification.
Currently, H5Spark parallelize the IO along the slowest dimension, i.e., the dimension that changes slowest on disks. For example, for a 2D C array x[10][200], the h5spark will choose the first dimension to partition, and then the maximum partition it can have is only 10, which is also the maximum degree of parallelism.
If we want to parallelize along any user-specified dimension, the current code needs a little bit modification.