Performance improvement

[AtiyahElsheikh]

potential performance improvement by a variety of ways:

• [x] Implementation of removeDeads (making use of the index of the deads)
• [x] getting rid of the use of global variables #20
• [ ] doX! with the exact subpopulation #21
  • [x] alive sub-population
  • [ ] male & female sub-populations
  • [ ] maintaining sub-populations with minimal efforts
• [x] MultiAgents.jl (existing potential type instability)
• [x] make outputs of simulation functions become an argument (of type struct) #29
• [x] improve implementation of allocation to other houses #23
  • [x] improve implementation of finding empty houses,
  • [x] investigate if it is worth to make related functions, e.g. select_house return only one type
  • [ ] caching pre-computations (e.g. weight towns). Similar stuffs have been done and improved performance
  • [ ] caching adjacent towns
  • [ ] caching town densities
  • [x] caching inst. prop. (slight improvement for large-size populations done only with birth function)
  • [ ] same with other functions
• [x] #37
• [x] #40
• [ ] #41
• [x] #44 (partially improved)
• [ ] #30 (partially improved)
• [ ] trying struct of arrays pattern
• [ ] noperson()::Person instead of nothing::Type{Nothing}
• [ ] algorithmic / technical
  • [ ] removing many agents / deads at once
  • [x] views of arrays instead of copies (tiny stuffs)
• [ ] conducing consistency checking and assertion along unit tests #60

[AtiyahElsheikh]

[ ] getting rid of the use of global variables Debug instructions within simulation functions  #20

When disabling the macros for doDeaths! performance improves between 10-25%. But checking the underlying code, it is possible that another issue is causing this performance drop.

[AtiyahElsheikh]

[x] MultiAgents.jl (existing potential type instability)

Employing version 0.4 of MultiAgents.jl brought 15-25% runtime and memory performance improvement

[AtiyahElsheikh]

• doX! with the exact subpopulation passing the exact sub-population to doX! #21

Initial efforts show that this brings significant improvement in memory consumption due to less temporary arrays are created each iteration.

[AtiyahElsheikh]

[ ] getting rid of the use of global variables Debug instructions within simulation functions  #20

Isolating debug and assumptions checking in separate code and at a level not extensively executed (e.g. dodeaths rather than deaths!) seem to impact the performance positively

[AtiyahElsheikh]

[ ] make outputs of simulation function become an argument (of type struct)

Functions s.a. doDeaths and doBirths create arrays of references every iteration. Avoiding this seems to significantly improve the memory usage and performance. This has been initially realised as described in #29.

[AtiyahElsheikh]

• Make type Person a concrete type #37

Corresponding fix improves memory allocation by c.a. 30%

[AtiyahElsheikh]

[ ] improve implementation of allocation to other houses Improving implementation o movePeopleToEmptyHouse #23

• [ ] improve implementation of finding empty houses,
• [ ] investigate if it is worth to make related functions, e.g. select_house return only one type

This part has led to significant improvement due to reduced usage of temporary arrays. For an idea, executing version 0.2.5 with an initial population of 5000 (i.e. 10000 man and women), performance looked as follows:

V0.2.5: 20.237322 seconds (154.70 M allocations: 7.828 GiB, 3.65% gc time)

After improvement

7.653553 seconds (37.11 M allocations: 2.194 GiB, 2.75% gc time)

The improvement was also due to replacement of memoization in doMarriages function.

[AtiyahElsheikh]

[ ] investigate if it is worth to make related functions, e.g. select_house return only one type

I am not sure if improving this has contributed to the performance improvement

[AtiyahElsheikh]

• the usage of memoization in do_marriages #40

As expected, it turned out that the more natural approach is more efficient in terms of memory and runtime. In this particular case, it seems memoization was consuming alot of storage (or may be it is not properly implemented).

[AtiyahElsheikh]

after Version 0.3

• speedup 25 % (w.r.t. Version 0.2.8, overall ~ 6x faster in comparison with V0)
• Memory allocation reduced (~380k instead of 24M V0.2.8, 290M V0.2, 400M V0)
• Storage usage reduced (90 MB instead of 10GB V0.2, 12.5 GB V0)