HSLdevcom / helmet-model-system

European Union Public License 1.2
10 stars 3 forks source link

Improve matrix memory consumption #495

Closed zptro closed 1 year ago

zptro commented 1 year ago

Emme uses float32 matrices by default. There is no reason for us to use float64 matrices in the demand model, because they will be transformed into float32 matrices in Emme assignment anyway.

AgentModelSystem previously utilized numpy.random.choice(), which required float64 matrices (otherwise it complained that the the probabilities did not sum to 1), but in #244 numpy.random.choice() was replaced with the much more efficient numpy.searchsorted(), which does not require float64 matrices.

If a parameter in destination choice is only for the own municipality or for all municipalities except the own, the variable array needs to be multiplied by a bool matrix. In ZoneData, the result of this multiplication has so far been stored for many kinds of variables. This PR changes the behavior so that the multiplication is done on the fly only when needed.

This makes memory consumption much smaller for large networks! 🚀