fmihpc / vlasiator

Vlasiator - ten letters you can count on
https://www.helsinki.fi/en/researchgroups/vlasiator
Other
45 stars 37 forks source link

Assigning block data to cells is inefficient at restart #173

Closed galfthan closed 9 years ago

galfthan commented 9 years ago

In _readBlockData in ioread.cpp the assignement of blockdata could be streamlined.

Now it is computing a velocity for each phase space cell, and adding that individually to the velocity mesh using the slow set_value operation (one hash map lookup and a lot of index math). Instead on can create all blocks in one operation and then copy all blockdata with one loop.

The number of phase space cells per process can be very large, up to 100 M, and thus this inefficiency may be visible in run-time. Most likely the actual file io is still the big bottleneck.

In https://github.com/galfthan/vlasiator/tree/faster_block_reads there is a patch, but it has not yet been tested. Once that has been done (performance and that it actually works) a pull request can be done.

galfthan commented 9 years ago

I checked with testpackage version of Magnetosphere that restarting with this patch is bitidentical compared to restarting with the old version

galfthan commented 9 years ago

Performance test on Voima. Restarting AAS_remake_pqm2 from 1149s (350GB file) on 60 nodes.

Performance, old above and new below

#                          avgtime             maxtime         mintime 
original: readBlockData    129.3     96.45     235.9     490   25.52     1     1 
withfix:  readBlockData    108.8     94.34     210.4     210   6.929     1     1 

Some performance improvement, even if this is so small that it might also be random fluctuation. One can also see that we have bad load balance when reading. Max is almost 2x average

galfthan commented 9 years ago

This is fixed