search

paulmillr / readdirp

Recursive version of fs.readdir with streaming api.
https://paulmillr.com
MIT License
378 stars 51 forks source link

highwatermark option #141

Closed ronag closed 4 years ago

ronag commented 4 years ago

highwatermark: 1 will negatively impact performance, but a larger value breaks a test scenario which depends on the number of elements buffered.

This fixes both.

paulmillr commented 4 years ago

highwatermark: 1 will negatively impact performance

I actually can't validate this. Try node examples/bench for highWaterMark: 1 and 16. It's the same for both cases +- 5%

ronag commented 4 years ago

I actually can't validate this. Try node examples/bench for highWaterMark: 1 and 16. It's the same for both cases +- 5%

Hm, it should happen if you have a folder containing lots of folders with no or few elements inside with a reader which reads in bursts with pauses between. It's a bit tricky to verify.

paulmillr commented 4 years ago

Understood.

paulmillr commented 4 years ago

@ronag is there any way to make highWaterMark work more? The #140 slows us down 1.5x from 3.2.0, and it isn't cool. As far as I understand, the work must continue until watermark is hit.

ronag commented 4 years ago

What benchmark?

paulmillr commented 4 years ago

node examples/bench on a dir of 200k files

ronag commented 4 years ago

How do I easily create a dir with 200k files?

ronag commented 4 years ago

@paulmillr: the way 3.2.0 achieves the performance is by having pretty much unlimited parallelism and a bit random ordering. My test with 300k files takes 1G memory, while master take 400M and a PR I'm working which is just slightly slower than master is 100M. I'll see if I can make this more configurable through the watermark.

master:
10000 rss=526.2M  heapTotal=450.3M  heapUsed=427.7M  external=0.8M   
1016000 rss=530.0M  heapTotal=455.0M  heapUsed=421.9M  external=0.8M   
1033000 rss=524.4M  heapTotal=457.4M  heapUsed=422.1M  external=0.8M   
1055000 rss=524.6M  heapTotal=457.4M  heapUsed=423.7M  external=0.8M   
1075000 rss=524.6M  heapTotal=457.4M  heapUsed=427.1M  external=0.8M   
1088000 rss=524.6M  heapTotal=457.4M  heapUsed=426.3M  external=0.8M   
1108000 rss=524.7M  heapTotal=457.4M  heapUsed=435.1M  external=0.8M   
1125000 rss=524.8M  heapTotal=457.4M  heapUsed=430.6M  external=0.8M   
1156000 rss=524.8M  heapTotal=457.4M  heapUsed=435.0M  external=0.8M   
1182000 rss=524.8M  heapTotal=457.4M  heapUsed=437.4M  external=0.8M   
1213000 rss=525.0M  heapTotal=457.4M  heapUsed=433.8M  external=0.8M   
1242000 rss=525.0M  heapTotal=457.4M  heapUsed=427.2M  external=0.8M   
1274000 rss=525.0M  heapTotal=457.4M  heapUsed=427.9M  external=0.8M   
1305000 rss=525.4M  heapTotal=457.9M  heapUsed=429.9M  external=0.8M   
1336000 rss=525.4M  heapTotal=457.9M  heapUsed=433.8M  external=0.8M   
1370000 rss=525.5M  heapTotal=457.9M  heapUsed=432.2M  external=0.8M   
1398000 rss=525.6M  heapTotal=458.5M  heapUsed=432.9M  external=0.8M   
1429000 rss=525.6M  heapTotal=458.5M  heapUsed=434.4M  external=0.8M   
1459000 rss=525.7M  heapTotal=458.5M  heapUsed=435.5M  external=0.8M   
1486000 rss=525.7M  heapTotal=458.5M  heapUsed=432.5M  external=0.8M   
1520000 rss=525.8M  heapTotal=458.5M  heapUsed=425.0M  external=0.8M   
1546000 rss=525.9M  heapTotal=458.5M  heapUsed=425.3M  external=0.8M   
1575000 rss=525.9M  heapTotal=458.5M  heapUsed=439.9M  external=0.8M   
1607000 rss=526.0M  heapTotal=458.5M  heapUsed=431.4M  external=0.8M   
1638000 rss=526.0M  heapTotal=458.5M  heapUsed=431.2M  external=0.8M   
1668000 rss=526.1M  heapTotal=458.5M  heapUsed=426.5M  external=0.8M   
1703000 rss=526.1M  heapTotal=458.5M  heapUsed=428.7M  external=0.8M   
1731000 rss=526.3M  heapTotal=459.2M  heapUsed=431.7M  external=0.8M   
1764000 rss=526.3M  heapTotal=459.2M  heapUsed=426.9M  external=0.8M   
1797000 rss=526.3M  heapTotal=459.2M  heapUsed=437.7M  external=0.8M   
1827000 rss=526.4M  heapTotal=459.2M  heapUsed=435.9M  external=0.8M   
1857000 rss=526.4M  heapTotal=459.2M  heapUsed=434.0M  external=0.8M   
1887000 rss=526.6M  heapTotal=459.5M  heapUsed=432.2M  external=0.8M   
1911000 rss=526.6M  heapTotal=459.5M  heapUsed=424.3M  external=0.8M   
1943000 rss=526.7M  heapTotal=459.5M  heapUsed=434.3M  external=0.8M   
1976000 rss=526.7M  heapTotal=459.5M  heapUsed=427.9M  external=0.8M   
2007000 rss=526.8M  heapTotal=459.5M  heapUsed=437.1M  external=0.8M   
2038000 rss=526.8M  heapTotal=459.5M  heapUsed=437.0M  external=0.8M   
2070000 rss=526.9M  heapTotal=459.5M  heapUsed=426.6M  external=0.8M   
2104000 rss=526.9M  heapTotal=459.5M  heapUsed=439.7M  external=0.8M   
2132000 rss=527.0M  heapTotal=459.5M  heapUsed=439.8M  external=0.8M   
2159000 rss=527.0M  heapTotal=459.5M  heapUsed=437.8M  external=0.8M   
2192000 rss=527.1M  heapTotal=459.5M  heapUsed=429.9M  external=0.8M   
2222000 rss=527.2M  heapTotal=459.8M  heapUsed=427.2M  external=0.8M   
2249000 rss=527.2M  heapTotal=459.8M  heapUsed=438.9M  external=0.8M   
2281000 rss=527.2M  heapTotal=459.8M  heapUsed=436.8M  external=0.8M   
2308000 rss=527.4M  heapTotal=460.6M  heapUsed=438.8M  external=0.8M   
2337000 rss=527.4M  heapTotal=460.6M  heapUsed=435.6M  external=0.8M   
2369000 rss=527.5M  heapTotal=460.6M  heapUsed=426.4M  external=0.8M   
2400000 rss=527.5M  heapTotal=460.6M  heapUsed=426.9M  external=0.8M   
2422000 rss=527.5M  heapTotal=460.6M  heapUsed=432.1M  external=0.8M   
2445000 rss=527.5M  heapTotal=460.6M  heapUsed=432.4M  external=0.8M   
2469000 rss=527.6M  heapTotal=460.6M  heapUsed=433.2M  external=0.8M   
2491000 rss=527.6M  heapTotal=460.6M  heapUsed=425.5M  external=0.8M   
2514000 rss=527.6M  heapTotal=460.6M  heapUsed=437.2M  external=0.8M   
2531000 rss=527.7M  heapTotal=460.6M  heapUsed=431.6M  external=0.8M   
2555000 rss=527.7M  heapTotal=460.6M  heapUsed=426.3M  external=0.8M   
2586000 rss=527.8M  heapTotal=460.6M  heapUsed=428.9M  external=0.8M   
2611592 rss=527.8M  heapTotal=460.6M  heapUsed=432.6M  external=0.8M   
Processed 2611592 files in 41840 msecs

3.2.0:
    0 rss=18.5M   heapTotal=4.7M    heapUsed=2.6M    external=0.8M   
 31000 rss=117.2M  heapTotal=69.5M   heapUsed=39.8M   external=0.8M   
 77000 rss=160.4M  heapTotal=90.2M   heapUsed=58.7M   external=0.8M   
135000 rss=240.7M  heapTotal=139.5M  heapUsed=110.0M  external=0.8M   
187000 rss=259.8M  heapTotal=144.0M  heapUsed=68.1M   external=0.8M   
238000 rss=264.1M  heapTotal=144.0M  heapUsed=101.4M  external=0.8M   
291000 rss=293.8M  heapTotal=169.9M  heapUsed=148.5M  external=0.8M   
346000 rss=338.3M  heapTotal=204.3M  heapUsed=177.1M  external=0.8M   
411000 rss=379.1M  heapTotal=243.3M  heapUsed=221.0M  external=0.8M   
454000 rss=769.9M  heapTotal=581.0M  heapUsed=550.4M  external=0.8M   
1475000 rss=771.2M  heapTotal=598.1M  heapUsed=571.2M  external=0.8M   
1534000 rss=808.6M  heapTotal=635.3M  heapUsed=600.2M  external=0.8M   
1614000 rss=849.0M  heapTotal=675.1M  heapUsed=642.5M  external=0.8M   
1670000 rss=881.8M  heapTotal=708.2M  heapUsed=672.6M  external=0.8M   
1721000 rss=914.1M  heapTotal=739.9M  heapUsed=712.7M  external=0.8M   
1775000 rss=948.4M  heapTotal=773.4M  heapUsed=751.7M  external=0.8M   
1838000 rss=983.9M  heapTotal=809.1M  heapUsed=779.8M  external=0.8M   
1894000 rss=1016.1M heapTotal=840.5M  heapUsed=807.2M  external=0.8M   
1971000 rss=1050.9M heapTotal=875.7M  heapUsed=854.2M  external=0.8M   
2034000 rss=1085.8M heapTotal=910.0M  heapUsed=883.6M  external=0.8M   
2092000 rss=1120.3M heapTotal=944.1M  heapUsed=919.3M  external=0.8M   
2143000 rss=1151.7M heapTotal=975.8M  heapUsed=945.9M  external=0.8M   
2197000 rss=1183.1M heapTotal=1007.0M heapUsed=972.9M  external=0.8M   
2271000 rss=1216.0M heapTotal=1039.8M heapUsed=1014.1M external=0.8M   
2330000 rss=1247.5M heapTotal=1071.2M heapUsed=1049.3M external=0.8M   
2381000 rss=1279.6M heapTotal=1102.7M heapUsed=1073.9M external=0.8M   
2448000 rss=1311.9M heapTotal=1134.7M heapUsed=1104.8M external=0.8M   
2505000 rss=1344.3M heapTotal=1167.2M heapUsed=1136.1M external=0.8M   
2563000 rss=1374.0M heapTotal=1196.8M heapUsed=1173.0M external=0.8M   
2610000 rss=1400.3M heapTotal=1223.0M heapUsed=1190.2M external=0.8M   
2611596 rss=1400.3M heapTotal=1223.0M heapUsed=1187.4M external=0.8M   
Processed 2611596 files in 20784 msecs