search

gem / oq-engine

OpenQuake Engine: a software for Seismic Hazard and Risk Analysis
https://github.com/gem/oq-engine/#openquake-engine
GNU Affero General Public License v3.0
382 stars 277 forks source link

The GMPEs for Canada are 10x slower than normal #7153

Closed micheles closed 2 years ago

micheles commented 3 years ago

See the example attached (job.zip). With the GMPE NBCC2015_AA13_activecrustFRjb_central "compute mean_std" on my workstation takes 1_462 seconds, while with the BooreAtkinson2008 only 152.6 seconds are needed. Profiling tells me that a lot of time is spent in gmpe_table.apply_magnitude_interpolation, so we should optimize there.

micheles commented 3 years ago

NB: all GMPETables are affected by this issue, most notably the NGAEast GMPEs, it is not just Canada.

g-weatherill commented 3 years ago

It would be good to see an optimisation here. One point of note that may be important is that the interpolations are applied in sequence: interpolate to the required period, then to the required magnitude then to the required distances. This always assumed that magnitude was scalar, which may now not be the case. The point to take note of is that although you are effectively slicing and interpolating from larger dimension arrays to lower dimension arrays, one cannot necessarily use N-dimensional interpolation (e.g. LinearNDInterpolator) as the units of the quantities in the different dimensions (T, M, R) are different and vary between log-scaling and linear scaling.

micheles commented 3 years ago

Probably the solution will be to remove the interpolations from the compute method and to perform them in the __init__ method by creating a customized lookup table.

g-weatherill commented 3 years ago

Well the interpolations are scenario dependent. From what I'm seeing, however, you are looping over the IMTs, meaning that it is interpolating to the required IMT on each loop (https://github.com/gem/oq-engine/blob/master/openquake/hazardlib/gsim/gmpe_table.py#L503). Perhaps that could be optimised by interpolating to the list of desired IMTs outside of the loop?

micheles commented 3 years ago

Not so easy. Anyway you can have a try at optimizing, if you want, the interesting example is the the one attached before in the job.zip file. Notice that for the moment this is low priority for us.

g-weatherill commented 3 years ago

This would be low priority for me too. I was just trying to make a suggestion to help with the optimisation. We have never benchmarked properly how long the GMPE tables take to run compared to the more conventional parameteric GMPEs, so a factor 10 difference is not as bad as I feared and I don't know what would be a "reasonable" benchmark performance-wise! If I find a time slot to work on it, I'll take a look.

micheles commented 2 years ago

After the change in https://github.com/gem/oq-engine/pull/7570 now "compute mean and stddevs" is 14x faster than in engine 3.13 and the Canada model is dominated by "get_poes", so there is no point in optimizing more. The case is closed.