search

NeurodataWithoutBorders / matnwb

A Matlab interface for reading and writing NWB files
BSD 2-Clause "Simplified" License
49 stars 32 forks source link

Accelerate iterative data writing through GPU power #507

Open GoktugAlkan opened 1 year ago

GoktugAlkan commented 1 year ago

Hello,

I am converting a large matrix with dimensions 261x252000000 to an NWB file. I am using the iterative data writing method. However, it takes a lot of time to finish the conversion. The computer where the large matrix is located and where we want to store the NWB file has a GPU.

Hence, I am wondering if GPU power can be used to accelerate this procedure. If this is not possible, could you suggest another approach to enhance the speed of the conversion?

Many thanks for considering my request!

oruebel commented 1 year ago

The limiting factor on performance for conversion is not compute power (i.e., CPU or GPU resources) but the I/O bandwidth. Generally speaking, in the iterative write I would try and increase the amount of data that is being written at-a-time, e.g., instead of writing 10MB per iteration try to write 1GB per iteration.

GoktugAlkan commented 1 year ago

Thanks @oruebel. Are there updates concerning writing data in parallel threads? I am referring to this discussion.