The tests contain logic that is able to catch large memory leaks by repeating certain operation N times and cleaning mem through garbage collection each time. As the mem leaks is iterrative process the success of approach will be determined by how large the mem leak is. For small mem leaks this process will not be able to capture them unless repeated many many times, which could be very costly in terms of time.
It is advised that when approach is taken and new test is prepared the check procedure parameters to be finetuned for that particular case.. To do so you need to execute the process from command line with "-s" option so that pytest does not hide the output of print statements. See more in the doc of the procedure..
The output that you should expect from precedure should be like:
tart check process for memory leaks
Num iterrations: 30
Maxumum memory growth/lost to the examined process: 4808662528
Maxumum machine memory utilization: 4808662528
Lets pause for 10 secs so GC to kick in
Available memory 19730.07MB/[20688478208]
Process initial RSS 8671.72MB/[9092956160]
Starting watched code ...........
Lets pause for 7 secs so GC to kick in
Iter No[0] : Process did added (or if - number means cleaned) 1014.14MB/[1063407616] . Avail memory: 18793.03MB/[19705921536] Used Mem: 61.0%
Overall stats : Process growth since start 1014.14MB/[1063407616] AVG growth per iter 1014.14MB/[1063407616.0]
Minimum growth so far: 1014.14MB/[1063407616]
Maximum growth so far: 1014.14MB/[1063407616]
Number of times there was 50% drop in memory: 0
Starting watched code ...........
Lets pause for 7 secs so GC to kick in
Iter No[1] : Process did added (or if - number means cleaned) 990.93MB/[1039065088] . Avail memory: 17372.98MB/[18216894464] Used Mem: 63.9%
Overall stats : Process growth since start 2005.07MB/[2102472704] AVG growth per iter 1002.54MB/[1051236352.0]
Minimum growth so far: 1014.14MB/[1063407616]
Maximum growth so far: 2005.07MB/[2102472704]
Number of times there was 50% drop in memory: 0
Checklist for code changes...
- [ ] Have you updated the relevant docstrings, documentation and copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)?
- [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?
Reference Issues/PRs
The tests contain logic that is able to catch large memory leaks by repeating certain operation N times and cleaning mem through garbage collection each time. As the mem leaks is iterrative process the success of approach will be determined by how large the mem leak is. For small mem leaks this process will not be able to capture them unless repeated many many times, which could be very costly in terms of time.
It is advised that when approach is taken and new test is prepared the check procedure parameters to be finetuned for that particular case.. To do so you need to execute the process from command line with "-s" option so that pytest does not hide the output of print statements. See more in the doc of the procedure..
The output that you should expect from precedure should be like:
tart check process for memory leaks Num iterrations: 30 Maxumum memory growth/lost to the examined process: 4808662528 Maxumum machine memory utilization: 4808662528 Lets pause for 10 secs so GC to kick in Available memory 19730.07MB/[20688478208]
Process initial RSS 8671.72MB/[9092956160] Starting watched code ........... Lets pause for 7 secs so GC to kick in Iter No[0] : Process did added (or if - number means cleaned) 1014.14MB/[1063407616] . Avail memory: 18793.03MB/[19705921536] Used Mem: 61.0% Overall stats : Process growth since start 1014.14MB/[1063407616] AVG growth per iter 1014.14MB/[1063407616.0] Minimum growth so far: 1014.14MB/[1063407616] Maximum growth so far: 1014.14MB/[1063407616] Number of times there was 50% drop in memory: 0 Starting watched code ........... Lets pause for 7 secs so GC to kick in Iter No[1] : Process did added (or if - number means cleaned) 990.93MB/[1039065088] . Avail memory: 17372.98MB/[18216894464] Used Mem: 63.9% Overall stats : Process growth since start 2005.07MB/[2102472704] AVG growth per iter 1002.54MB/[1051236352.0] Minimum growth so far: 1014.14MB/[1063407616] Maximum growth so far: 2005.07MB/[2102472704] Number of times there was 50% drop in memory: 0
..........................................................
What does this implement or fix?
Any other comments?
Checklist
Checklist for code changes...
- [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?