search

mdahirel / pushed-pulled-2020-heritability-IBM

code for the paper "Individual variation in dispersal, and its sources, shape the fate of pushed vs. pulled range expansions"
0 stars 0 forks source link

data wrangling slower than it used to #25

Closed mdahirel closed 3 years ago

mdahirel commented 3 years ago

The parts of the analyse_simulations code where we build the tab_intermediate and tab_front tables seem to be way slower than I remember them too (seem, because I never timed the code, so I may misremember; but I'm almost sure).

After a quick check the bottlenecks seem to be at the level of the mutate() and summarise() calls there (too many groups? too many variables to summarise?)

Possible solution: use map() to avoid having everything in memory at the same time?

mdahirel commented 3 years ago

Bottleneck for tab_intermediate solved (by moving all operations I could outside of the group_by) as of https://github.com/mdahirel/pushed-pulled-2020-heritability-IBM/commit/da617ccba5a32a19abf760307284eb48b4ababfc

mdahirel commented 3 years ago

Made some changes that slightly improve speed, but I think I can't go much faster without working a lot on it/ changing completely the style of this part of the code, and the benefits-costs is not worth it/ it risks making the code harder to read.

So closing the issue