search

yakra / tmtools

Tools to aid in development of the TravelMapping project
0 stars 0 forks source link

DBFtrim: speed tests #2

Closed yakra closed 6 years ago

yakra commented 6 years ago

Conduct speed tests of multi-file-crawl vs single-file-crawl versions

• with and without progress indicators

yakra commented 6 years ago

apples to apples comparison may not be possible, depending on swap file usage

for comparison purposes, using 20171030e-pb0td, a modified version of 20171030e with: • ProgBar disabled • timestamp enabled • unused value deleted ...compared against 20171031f with timestamp enabled

yakra commented 6 years ago 
test                multiple    ProgBar1A   ProgBar5    single
----------------------------------------------------------------------------------------
Road_Inventory (“both2”)    00:00:31    00:01:20    00:00:20    00:00:08
Road_Inventory.Ytrim        00:01:54    00:01:30    00:00:30    00:00:22
Road_Inventory.Ycull        00:01:37    00:01:46    00:00:55    00:00:55
Road_Inventory.Yorig        00:10:26    00:03:07    00:02:40    00:02:27

I may end up using ProgBar5 going forward, or possibly a variant (such as ProgBar7)...

yakra commented 6 years ago

Test with progress indicators when saving the trimmed file, in addition to those when scanning the original file.

yakra commented 6 years ago 
test                ProgBar1A x2    ProgBar5 x2
-----------------------------------------------------------
Road_Inventory (“both2”)    00:02:27    00:00:29
Road_Inventory.Ytrim        00:02:27    00:00:30
Road_Inventory.Ycull        00:02:27    00:00:59
Road_Inventory.Yorig        00:02:56    00:02:44
yakra commented 6 years ago

Conclusions: • ProgBar1A greatly increases runtime for smaller files, such that execution takes longer than the "multiple crawl" version, often with only a single ProgBar. Not worth using. • ProgBar5 can significantly increase runtime on smaller files, but overall runtime is still acceptable. On larger files, the performance hit scales downward; execution time doesn't increase by as great a percentage. • A second ProgBar5 adds little additional performance hit.

I'll most likely be using DBFtrim on larger files. Having some visual indication of how far along it is in the process would be a Good Thing. Two ProgBar5s it is.