search

python / asyncio

asyncio historical repository
https://docs.python.org/3/library/asyncio.html
1.04k stars 177 forks source link

Use bytearray for buffer in _UnixWritePipeTransport #385

Closed methane closed 8 years ago

methane commented 8 years ago

fixes #384

This pull request is a control experiment of #339.

methane commented 8 years ago

benchmark: draintest.py

master:

DEBUG:asyncio:Using selector: KqueueSelector
1e+07 0.8767850399017334
2e+07 8.521645069122314
4e+07 34.822274923324585
^C

socketpair:writev (#339)

DEBUG:asyncio:Using selector: KqueueSelector
1e+07 0.028395891189575195
2e+07 0.049124956130981445
4e+07 0.09378409385681152
1e+08 0.24453306198120117
2e+08 0.5015709400177002
4e+08 0.9511210918426514

methane:unix-events-bytearray-buffer

DEBUG:asyncio:Using selector: KqueueSelector
1e+07 0.0306088924407959
2e+07 0.057476043701171875
4e+07 0.11026811599731445
1e+08 0.27896881103515625
2e+08 0.5629270076751709
4e+08 1.111664056777954
methane commented 8 years ago

Another benchmark draintest2.py

339 :

$ time python3 draintest2.py
DEBUG:asyncio:Using selector: KqueueSelector
1e+07 0.10091304779052734
2e+07 0.25542306900024414
4e+07 0.5010809898376465
1e+08 1.2510888576507568
2e+08 2.4644618034362793
4e+08 4.908963203430176

real    0m9.668s
user    0m6.515s
sys 0m3.689s

this pull request:

$ time python3 draintest2.py
DEBUG:asyncio:Using selector: KqueueSelector
1e+07 0.04972386360168457
2e+07 0.10954093933105469
4e+07 0.20995807647705078
1e+08 0.5249450206756592
2e+08 1.0876190662384033
4e+08 2.0913729667663574

real    0m4.258s
user    0m3.024s
sys 0m1.657s
methane commented 8 years ago

More combination of chunksize. draintest3.py

339 :

DEBUG:asyncio:Using selector: KqueueSelector
10000 * 1000 0.1287369728088379
20000 * 1000 0.2537820339202881
40000 * 1000 0.49232983589172363
100000 * 1000 1.2934849262237549
200000 * 1000 2.859189033508301
400000 * 1000 4.890531063079834
1000 * 10000 0.05991005897521973
2000 * 10000 0.1615598201751709
4000 * 10000 0.3714721202850342
10000 * 10000 1.0015809535980225
20000 * 10000 2.029129981994629
40000 * 10000 4.136893033981323
100 * 100000 0.02856612205505371
200 * 100000 0.0627448558807373
400 * 100000 0.12400102615356445
1000 * 100000 0.49852800369262695
2000 * 100000 1.5194988250732422
4000 * 100000 3.5665440559387207

real    0m23.664s
user    0m14.943s
sys 0m10.145s

this pull request:

DEBUG:asyncio:Using selector: KqueueSelector
10000 * 1000 0.05972003936767578
20000 * 1000 0.10161590576171875
40000 * 1000 0.22113490104675293
100000 * 1000 0.5041630268096924
200000 * 1000 1.0610871315002441
400000 * 1000 2.1884210109710693
1000 * 10000 0.03620195388793945
2000 * 10000 0.07513689994812012
4000 * 10000 0.1556410789489746
10000 * 10000 0.39647483825683594
20000 * 10000 0.7405750751495361
40000 * 10000 1.5679919719696045
100 * 100000 0.05654788017272949
200 * 100000 0.10537409782409668
400 * 100000 0.18746709823608398
1000 * 100000 0.43062806129455566
2000 * 100000 0.9680559635162354
4000 * 100000 1.6960680484771729

real    0m10.811s
user    0m6.724s
sys 0m5.137s

So, when chunk is small (~10KB), bytearray is faster always.

When chunk is large (100KB), deque [bytes] is faster. But bytearray overtakes when .write() called more than 1000 times.

methane commented 8 years ago

@socketpair Could you confirm draintest3.py bench?

socketpair commented 8 years ago

@methane

Could you confirm draintest3.py bench?

what does that mean ?

methane commented 8 years ago

@socketpair I meant could you confirm draintest3.py measures right thing, and benchmark in you environment. FYI, original draintest.py measures only .drain(). It doesn't include .write(data) time. So it was not good for benchmarking bytearray vs deque[bytes].

1st1 commented 8 years ago

Committed in 6f8f833cca6fa414e77c8742ffadf01bcf6b24f1. Thank you!