yakra / tmtools

Tools to aid in development of the TravelMapping project
0 stars 0 forks source link

DBFtrim: delete unused field data values #3

Closed yakra closed 6 years ago

yakra commented 6 years ago

• delete unused values that don't get assigned to MaxVal • check RAM usage. Fixed in eb65347b6c90d2b956b56518753fd0e74d6ea0d8

yakra commented 6 years ago

single-crawl vs multi-crawl RAM usage test on a fresh-rebooted system

yakra commented 6 years ago

Baseline

Mem:    695.5M  17.6%   3.8 G   Swap:   0

Road_Inventory.dbf (115.6 MB, "both2" version)

• Before, multi:    Mem:    703.8M  17.9%   3.8G    Swap:   0
• Before, single:   Mem:    673.5M  17.1%   3.8G    Swap:   0
• After, single:    Mem:    663.4M  16.8%   3.8G    Swap:   0

Road_Inventory.Ytrim.dbf (264.5 MB)

• Before, multi:    Mem:    711.0M  18.0%   3.8G    Swap:   0
• Before, single:   Mem:    665.0M  16.9%   3.8G    Swap:   0
• After, single:    Mem:    703.6M  17.8%   3.8G    Swap:   0

Road_Inventory.Ycull.dbf (890.8 MB)

• Before, multi:    Mem:    697.0M  17.7%   3.8G    Swap:   0
• Before, single:   Mem:    699.4M  17.7%   3.8G    Swap:   0
• After, single:    Mem:    702.2M  17.8%   3.8G    Swap:   0

Road_Inventory.Yorig.dbf (2.6 GB)

• Before, multi:    Mem:    425.3M  10.8%   3.8G    Swap:   326.3M  8.0%    4.0G
• Before, single:   Mem:    446.1M  14.1%   3.8G    Swap:   247.8M  6.1%    4.0G
• After, single:    Mem:    704.2M  17.9%   3.8G    Swap:   0
yakra commented 6 years ago

The figures above don't tell the full story, particularly how for Yorig, memory usage steadily climbed to almost 100% during the Before tests, then stayed practically flat in the After tests. With the exception of "Before, multi", the amount of memory (mem+swap) used is about the same across the board. Alt+Tabbing to Pluma after completion seems to immediately add ~15 MB, so there's some more noise added to the usual fluctuations...

In any case: Fixed in eb65347. Further deletions of unused variables can get a separate issue. Or not.

yakra commented 6 years ago

15b14c14e39088ae5f395b7508a8d8c2d6f8b52e also relevant