search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
138 stars 51 forks source link

make it possible to manually set chunks when loading dask arrays #1477

Closed pordyna closed 1 year ago

pordyna commented 1 year ago

Make it possible to override the default chunking behavior when loading to dask arrays. For PIConGPU the default creates a chunk for each GPU with its local domain. When using static load balancing with gridDist in PIConGPU, this results in very uneven chunks. It looks like, for many operations, it makes sense to chunk along the outermost direction only (reading contiguous domains is faster). With this PR this can be achieved with .to_daks_array(chunks={0 : 'auto', 1: -1, 2: -1})

pordyna commented 1 year ago

Thanks @ax3l, I applied your suggestions.