search

paulmillr / readdirp

Recursive version of fs.readdir with streaming api.
https://paulmillr.com
MIT License
378 stars 51 forks source link

more parallelism and simplification #143

Closed ronag closed 4 years ago

ronag commented 4 years ago

WIP! Don't merge.

@paulmillr Would you mind running your benchmark against this PR and compare with master & 3.2.0

Here are my results:

PR:  
2611598 rss=212.9M  heapTotal=150.0M  heapUsed=123.4M  external=0.8M   
Processed 2611598 files in 28196 msecs

master:  
2611592 rss=527.8M  heapTotal=460.6M  heapUsed=432.6M  external=0.8M   
Processed 2611592 files in 41840 msecs

3.2.0:  
2611596 rss=1400.3M heapTotal=1223.0M heapUsed=1187.4M external=0.8M   
Processed 2611596 files in 20784 msecs
ronag commented 4 years ago

Another run with a different test set:

PR: 22.6M @ 21365 ms 3.2.0: 301.2M @ 14191 ms

1,5x time at 0,075x memory, seems like a decent tradeoff?

PR
    0 rss=18.5M   heapTotal=4.7M    heapUsed=2.6M    external=0.8M   
 26000 rss=44.0M   heapTotal=24.7M   heapUsed=9.1M    external=0.8M   
 57000 rss=45.1M   heapTotal=25.5M   heapUsed=12.3M   external=0.8M   
 78000 rss=60.6M   heapTotal=42.0M   heapUsed=16.5M   external=0.8M   
105000 rss=63.3M   heapTotal=42.0M   heapUsed=9.7M    external=0.8M   
134000 rss=63.3M   heapTotal=42.0M   heapUsed=15.0M   external=0.8M   
177000 rss=63.5M   heapTotal=42.0M   heapUsed=6.1M    external=0.8M   
214000 rss=63.7M   heapTotal=42.0M   heapUsed=17.2M   external=0.8M   
254000 rss=63.7M   heapTotal=42.0M   heapUsed=17.4M   external=0.8M   
299000 rss=63.8M   heapTotal=42.0M   heapUsed=11.4M   external=0.8M   
339000 rss=64.0M   heapTotal=42.3M   heapUsed=10.9M   external=0.8M   
380000 rss=64.3M   heapTotal=42.3M   heapUsed=12.6M   external=0.8M   
421000 rss=64.7M   heapTotal=43.6M   heapUsed=9.7M    external=0.8M   
463000 rss=64.1M   heapTotal=42.0M   heapUsed=13.4M   external=0.8M   
499000 rss=64.2M   heapTotal=42.0M   heapUsed=13.6M   external=0.8M   
533000 rss=64.3M   heapTotal=42.0M   heapUsed=17.5M   external=0.8M   
568000 rss=64.3M   heapTotal=42.0M   heapUsed=15.4M   external=0.8M   
608000 rss=64.3M   heapTotal=42.0M   heapUsed=16.6M   external=0.8M   
653000 rss=64.5M   heapTotal=42.3M   heapUsed=18.7M   external=0.8M   
700000 rss=64.7M   heapTotal=42.3M   heapUsed=13.6M   external=0.8M   
738000 rss=64.9M   heapTotal=42.3M   heapUsed=16.4M   external=0.8M   
783000 rss=65.0M   heapTotal=43.3M   heapUsed=19.4M   external=0.8M   
825000 rss=66.0M   heapTotal=43.8M   heapUsed=13.6M   external=0.8M   
866000 rss=66.3M   heapTotal=43.8M   heapUsed=13.9M   external=0.8M   
900000 rss=67.2M   heapTotal=45.2M   heapUsed=21.0M   external=0.8M   
940000 rss=67.7M   heapTotal=45.4M   heapUsed=25.2M   external=0.8M   
983000 rss=68.5M   heapTotal=46.2M   heapUsed=21.2M   external=0.8M   
1028000 rss=64.9M   heapTotal=42.5M   heapUsed=4.6M    external=0.8M   
1069000 rss=64.9M   heapTotal=42.5M   heapUsed=6.8M    external=0.8M   
1112000 rss=64.9M   heapTotal=42.5M   heapUsed=15.2M   external=0.8M   
1154000 rss=64.9M   heapTotal=42.5M   heapUsed=16.0M   external=0.8M   
1198000 rss=65.1M   heapTotal=42.5M   heapUsed=13.5M   external=0.8M   
1237000 rss=65.1M   heapTotal=42.5M   heapUsed=9.7M    external=0.8M   
1278000 rss=65.2M   heapTotal=42.8M   heapUsed=11.5M   external=0.8M   
1313000 rss=65.5M   heapTotal=42.8M   heapUsed=23.9M   external=0.8M   
1352000 rss=65.5M   heapTotal=42.8M   heapUsed=21.7M   external=0.8M   
1397000 rss=66.0M   heapTotal=44.4M   heapUsed=16.6M   external=0.8M   
1437000 rss=66.4M   heapTotal=44.4M   heapUsed=17.4M   external=0.8M   
1471000 rss=66.8M   heapTotal=44.6M   heapUsed=21.6M   external=0.8M   
1505000 rss=66.9M   heapTotal=44.6M   heapUsed=19.3M   external=0.8M   
1534000 rss=67.3M   heapTotal=45.4M   heapUsed=11.1M   external=0.8M   
1568000 rss=68.0M   heapTotal=45.7M   heapUsed=22.6M   external=0.8M   
1611000 rss=65.1M   heapTotal=42.5M   heapUsed=5.0M    external=0.8M   
1611657 rss=65.1M   heapTotal=42.5M   heapUsed=8.2M    external=0.8M   
Processed 1611657 files in 21365 msecs

master
    0 rss=18.6M   heapTotal=4.7M    heapUsed=2.6M    external=0.8M   
 15000 rss=34.0M   heapTotal=14.5M   heapUsed=5.3M    external=0.8M   
 32000 rss=45.0M   heapTotal=25.8M   heapUsed=9.7M    external=0.8M   
 55000 rss=45.5M   heapTotal=26.0M   heapUsed=8.2M    external=0.8M   
 75000 rss=45.7M   heapTotal=26.0M   heapUsed=12.8M   external=0.8M   
 87000 rss=45.7M   heapTotal=26.0M   heapUsed=15.5M   external=0.8M   
104000 rss=45.8M   heapTotal=26.0M   heapUsed=12.1M   external=0.8M   
114000 rss=45.9M   heapTotal=26.0M   heapUsed=8.5M    external=0.8M   
126000 rss=45.9M   heapTotal=26.0M   heapUsed=15.1M   external=0.8M   
157000 rss=63.1M   heapTotal=42.8M   heapUsed=19.7M   external=0.8M   
183000 rss=63.1M   heapTotal=42.8M   heapUsed=22.9M   external=0.8M   
213000 rss=63.3M   heapTotal=42.8M   heapUsed=14.8M   external=0.8M   
242000 rss=63.3M   heapTotal=42.8M   heapUsed=25.0M   external=0.8M   
273000 rss=63.3M   heapTotal=42.8M   heapUsed=19.6M   external=0.8M   
303000 rss=61.4M   heapTotal=40.7M   heapUsed=11.2M   external=0.8M   
334000 rss=61.4M   heapTotal=40.7M   heapUsed=13.7M   external=0.8M   
368000 rss=61.4M   heapTotal=40.7M   heapUsed=12.0M   external=0.8M   
396000 rss=61.4M   heapTotal=40.7M   heapUsed=13.4M   external=0.8M   
427000 rss=61.4M   heapTotal=40.7M   heapUsed=14.5M   external=0.8M   
456000 rss=61.4M   heapTotal=40.7M   heapUsed=9.8M    external=0.8M   
486000 rss=61.5M   heapTotal=40.7M   heapUsed=5.8M    external=0.8M   
520000 rss=61.5M   heapTotal=40.7M   heapUsed=13.7M   external=0.8M   
546000 rss=61.5M   heapTotal=40.7M   heapUsed=12.7M   external=0.8M   
573000 rss=61.5M   heapTotal=40.7M   heapUsed=17.9M   external=0.8M   
601000 rss=61.5M   heapTotal=40.7M   heapUsed=6.8M    external=0.8M   
632000 rss=61.5M   heapTotal=40.7M   heapUsed=7.8M    external=0.8M   
663000 rss=61.5M   heapTotal=40.7M   heapUsed=7.1M    external=0.8M   
699000 rss=61.5M   heapTotal=40.7M   heapUsed=15.0M   external=0.8M   
727000 rss=61.5M   heapTotal=40.7M   heapUsed=17.0M   external=0.8M   
758000 rss=61.5M   heapTotal=40.7M   heapUsed=3.9M    external=0.8M   
792000 rss=61.5M   heapTotal=40.7M   heapUsed=17.4M   external=0.8M   
822000 rss=61.5M   heapTotal=40.7M   heapUsed=15.3M   external=0.8M   
851000 rss=61.7M   heapTotal=40.7M   heapUsed=10.4M   external=0.8M   
882000 rss=61.7M   heapTotal=40.7M   heapUsed=12.1M   external=0.8M   
908000 rss=61.7M   heapTotal=40.7M   heapUsed=14.1M   external=0.8M   
939000 rss=61.7M   heapTotal=40.7M   heapUsed=19.2M   external=0.8M   
972000 rss=61.7M   heapTotal=40.7M   heapUsed=12.4M   external=0.8M   
1002000 rss=61.7M   heapTotal=40.7M   heapUsed=15.8M   external=0.8M   
1034000 rss=61.7M   heapTotal=40.7M   heapUsed=5.9M    external=0.8M   
1066000 rss=61.8M   heapTotal=40.7M   heapUsed=12.4M   external=0.8M   
1100000 rss=61.8M   heapTotal=40.7M   heapUsed=9.4M    external=0.8M   
1128000 rss=61.8M   heapTotal=40.7M   heapUsed=8.6M    external=0.8M   
1155000 rss=61.8M   heapTotal=40.7M   heapUsed=7.5M    external=0.8M   
1189000 rss=61.8M   heapTotal=40.7M   heapUsed=20.9M   external=0.8M   
1219000 rss=61.8M   heapTotal=40.7M   heapUsed=17.8M   external=0.8M   
1247000 rss=61.8M   heapTotal=40.7M   heapUsed=10.3M   external=0.8M   
1279000 rss=61.8M   heapTotal=40.7M   heapUsed=16.6M   external=0.8M   
1307000 rss=61.8M   heapTotal=40.7M   heapUsed=6.3M    external=0.8M   
1335000 rss=61.8M   heapTotal=40.7M   heapUsed=16.6M   external=0.8M   
1367000 rss=61.8M   heapTotal=40.7M   heapUsed=6.6M    external=0.8M   
1398000 rss=61.9M   heapTotal=41.0M   heapUsed=6.5M    external=0.8M   
1420000 rss=61.9M   heapTotal=41.0M   heapUsed=18.4M   external=0.8M   
1444000 rss=61.9M   heapTotal=41.0M   heapUsed=13.4M   external=0.8M   
1468000 rss=61.9M   heapTotal=41.0M   heapUsed=13.7M   external=0.8M   
1489000 rss=61.9M   heapTotal=41.0M   heapUsed=16.4M   external=0.8M   
1512000 rss=62.0M   heapTotal=41.0M   heapUsed=12.4M   external=0.8M   
1530000 rss=62.0M   heapTotal=41.0M   heapUsed=12.1M   external=0.8M   
1555000 rss=62.0M   heapTotal=41.0M   heapUsed=11.7M   external=0.8M   
1586000 rss=62.1M   heapTotal=41.0M   heapUsed=16.0M   external=0.8M   
1611657 rss=62.1M   heapTotal=41.0M   heapUsed=19.4M   external=0.8M   
Processed 1611657 files in 30199 msecs

3.2.0
    0 rss=18.5M   heapTotal=4.7M    heapUsed=2.6M    external=0.8M   
 28000 rss=85.7M   heapTotal=57.0M   heapUsed=26.7M   external=0.8M   
 86000 rss=111.4M  heapTotal=74.3M   heapUsed=38.9M   external=0.8M   
146000 rss=154.2M  heapTotal=109.6M  heapUsed=86.7M   external=0.8M   
192000 rss=159.5M  heapTotal=112.4M  heapUsed=67.7M   external=0.8M   
251000 rss=185.9M  heapTotal=136.0M  heapUsed=112.0M  external=0.8M   
306000 rss=255.2M  heapTotal=188.7M  heapUsed=166.8M  external=0.8M   
340000 rss=256.0M  heapTotal=183.7M  heapUsed=89.7M   external=0.8M   
389000 rss=258.9M  heapTotal=183.7M  heapUsed=115.7M  external=0.8M   
461000 rss=263.3M  heapTotal=187.4M  heapUsed=162.3M  external=0.8M   
521000 rss=305.7M  heapTotal=228.0M  heapUsed=202.9M  external=0.8M   
578000 rss=347.9M  heapTotal=268.7M  heapUsed=235.1M  external=0.8M   
635000 rss=383.9M  heapTotal=303.0M  heapUsed=277.7M  external=0.8M   
683000 rss=413.0M  heapTotal=332.1M  heapUsed=298.3M  external=0.8M   
720000 rss=339.2M  heapTotal=258.2M  heapUsed=94.1M   external=0.8M   
796000 rss=340.0M  heapTotal=258.2M  heapUsed=124.9M  external=0.8M   
859000 rss=340.1M  heapTotal=258.2M  heapUsed=169.7M  external=0.8M   
919000 rss=340.6M  heapTotal=258.5M  heapUsed=201.8M  external=0.8M   
972000 rss=344.2M  heapTotal=262.9M  heapUsed=226.7M  external=0.8M   
1032000 rss=377.3M  heapTotal=295.9M  heapUsed=268.6M  external=0.8M   
1112000 rss=408.4M  heapTotal=326.9M  heapUsed=301.2M  external=0.8M   
1166000 rss=339.1M  heapTotal=256.9M  heapUsed=56.3M   external=0.8M   
1226000 rss=339.2M  heapTotal=256.9M  heapUsed=93.1M   external=0.8M   
1290000 rss=339.2M  heapTotal=256.9M  heapUsed=135.0M  external=0.8M   
1357000 rss=339.5M  heapTotal=257.1M  heapUsed=166.1M  external=0.8M   
1432000 rss=339.2M  heapTotal=257.1M  heapUsed=205.2M  external=0.8M   
1492000 rss=257.4M  heapTotal=175.4M  heapUsed=58.8M   external=0.8M   
1557000 rss=257.4M  heapTotal=175.4M  heapUsed=94.7M   external=0.8M   
1611655 rss=128.8M  heapTotal=57.4M   heapUsed=27.2M   external=0.8M   
Processed 1611655 files in 14191 msecs
paulmillr commented 4 years ago

@ronag The tests are failing. Could you check this?

ronag commented 4 years ago

Fixed test

paulmillr commented 4 years ago

Thank you.