Writing data is horribly slow

[qxcv]

It's not even get_stacks() that's slow any more, it's just writing to the HDF5 file. It seems to be taking much longer than it did before I had smaller subposes, so it could have something to do with all of the extra datasets which the HDF5 file needs to handle (maybe there's some shuffling going on?). The time to write also seems to increase rapidly with file size, which is worrying.

Some ideas for improving performance:

• Increase chunk size, particularly for reasonably sparse data (classes, limb positions).
• After the get_stacks() parfor, try writing all gathered data at once. Right now there is a single call to store3hdf6 for each individual stack, which means that the file needs to be opened, written to (extending datasets as necessary) and closed hundreds of times for each batch (obviously very slow!).
• Try writing data in background so that write time is not wasted. I can think of two strategies for that:
  1. After the parfor put the data in the right format sequentially, but do the actual writing in a parfeval; don't launch another writer task until the previous parfeval has finished (so you will need blocking calls after the parfor and after the sequential loop).
  2. Manually spawn all workers yourself using parfeval. That solution would require a lot of effort, but would also maximise throughput. IDK whether it's worth it :pensive:

[qxcv]

This is fixed for now (and I didn't even have to resort to crazy parallel writing hacks!). The problem was entirely in the number of calls to HDF5 wrapper routines (especially h5info), so batching a whole lot data up at once and calling store3hdf6 once per batch made things much faster.

I expect that this could become a problem again in the future, as it seems that the HDF5 C library has a bug where continually re-opening a file makes access much slower, but at least now it will take far longer for that problem to manifest itself.