moeyensj / thor

Tracklet-less Heliocentric Orbit Recovery
BSD 3-Clause "New" or "Revised" License
41 stars 14 forks source link

Improve peak memory by pushing sorting of large tables upstream of functions #148

Closed akoumjian closed 9 months ago

akoumjian commented 10 months ago

Large tables being sorted inside functions doubles the peak memory required, even if the parent caller is replacing the original table with some version of the sorted one. This is because the python garbage collector is not smart enough to know that the higher level reference is going to be replaced and a table sort duplicates the data (the same goes for defragmentation).

The next area to look at here is the test orbit ephemeris generation, as there are lots of id sorting and comparisons done. I think with some smart adjustments we may be able to reduce the peak memory there as well.