Differences in memory consumption between version 3.4.0 & 3.4.2

[zhangzju]

Details

Hi developers, I observed significant differences in memory consumption when performing the same calculations using different versions of abacus in similar environments.

Specifically, when I use version 3.4.0 of abacus, the memory consumption of all calculation processes is relatively stable and not high. When the number of processes is 32 (equal to the number of CPU cores), the average memory consumption is around 1.2GB. However, when I run the desired calculation with version 3.4.2 of abacus, this value keeps increasing and reaches around 3GB at generation 13. Finally, around generation 23, the process exits abnormally due to insufficient RSS. Related pics are as follows:

Detail version Normal Commit: 923b69739 (Wed Oct 4 08:03:06 2023 +0800) Abnormal Commit: 7670bb0 (Mon Nov 13 14:24:03 2023 +0800)

Result in 3.4.0

RSS in 3.4.0:

Result in 3.4.2:

RSS in 3.4.2:

Enviroment information(32c 128g):

Task list for Issue attackers (only for developers)

• [X] Reproduce the performance issue on a similar system or environment.
• [ ] Identify the specific section of the code causing the performance issue.
• [ ] Investigate the issue and determine the root cause.
• [ ] Research best practices and potential solutions for the identified performance issue.
• [ ] Implement the chosen solution to address the performance issue.
• [ ] Test the implemented solution to ensure it improves performance without introducing new issues.
• [X] Optimize the solution if necessary, considering trade-offs between performance and other factors (e.g., code complexity, readability, maintainability).
• [X] Review and incorporate any relevant feedback from users or developers.
• [ ] Merge the improved solution into the main codebase and notify the issue reporter.

[mohanchen]

Thanks! We suggest the issues be raised in another repository, https://github.com/deepmodeling/abacus-develop/issues