search

statnet / ergm

Fit, Simulate and Diagnose Exponential-Family Models for Networks
Other
98 stars 37 forks source link

add control argument to skip hashing in ergm_state #474

Closed chad-klumb closed 2 years ago

chad-klumb commented 2 years ago

as introduced in https://github.com/statnet/ergm/commit/0424bd80c3224e37b5e5a8e0cef61ae60e923607

Generating the hash can take a lot of time for some EpiModel applications.

cf. https://github.com/EpiModel/EpiModelHIV-p/issues/120

smjenness commented 2 years ago

Hi @krivit : could I ask that you prioritize this for the next ergm release? It has turned ~30 min EpiModel jobs into ~3 hour jobs. Thanks!

krivit commented 2 years ago

I see. It speeds up most parallel applications by quite a lot, since it only needs to serialise model and proposal objects once per ergm() call, but if you are running everything in a single thread, indeed there is no point in computing the hash.

krivit commented 2 years ago

Well, twice per ergm() call: once to hash, once to transmit.

smjenness commented 2 years ago

@krivit : any chance you could work on this for EpiModel/MIDAS time in the next week? It is an important fix to address for EpiModel work. Thanks!

krivit commented 2 years ago

It turns out to be nontrivial to put a control parameter into that code path. It may make sense to use an options() style option instead.

Perhaps if it's set to disable hashing, we can use a random string instead, which will save time on some use cases.

krivit commented 2 years ago

A possible fix is in the i474-state-hash-random branch. Please test.

AdrienLeGuillou commented 2 years ago

Thanks @krivit,

with this branch I get similar results with this code as I did with this commit. So it seems that's fixing it for me.

I will run full simulations on the HPC with it to make sure .

AdrienLeGuillou commented 2 years ago

Everything seems as fast as before with this branch, thanks @krivit