search

apple / swift-nio-http2

HTTP/2 support for SwiftNIO
https://swiftpackageindex.com/apple/swift-nio-http2/main/documentation/niohttp2
Apache License 2.0
465 stars 82 forks source link

HPACKHeaders performance test for HTTPHeader normalisation #346

Closed glbrntt closed 2 years ago

glbrntt commented 2 years ago

Motivation:

HPACKHeaders has an init which will normalize HTTPHeaders, this includes removing any headers which are illegal in HTTP2 and any headers which present as a 'connection' header value. As written this is expensive in some edges cases. With perf tests we can validate that implementation changes improve performance.

Modifications:

Add a performance test to normalize HTTPHeaders. This is run on two configurations.

Result:

More insight into header normalisation perf.

glbrntt commented 2 years ago

@swift-nio-bot test perf please

swift-server-bot commented 2 years ago

performance report

build id: 15

timestamp: Wed Sep 29 12:55:11 UTC 2021

results

nameminmaxmeanstd
1_conn_10k_reqs 1.6213979721069336 1.6442910432815552 1.632042407989502 0.006699844032143084
encode_100k_header_blocks_indexable 0.14471304416656494 0.14747297763824463 0.14572598934173583 0.0008046368602636831
encode_100k_header_blocks_nonindexable 0.3090909719467163 0.3160330057144165 0.3103432893753052 0.00211925033167702
encode_100k_header_blocks_neverIndexed 0.31516897678375244 0.31856203079223633 0.31606420278549197 0.0011066502001272586
decode_100k_header_blocks_indexable 0.11055600643157959 0.11164200305938721 0.11097080707550049 0.0003699525350451268
decode_100k_header_blocks_nonindexable 0.13858795166015625 0.13942408561706543 0.13902969360351564 0.0002693567257094209
decode_100k_header_blocks_neverIndexed 0.13593900203704834 0.13771402835845947 0.13646239042282104 0.00048347271333613114
hpackheaders_canonical_form 0.031477928161621094 0.032000064849853516 0.03155630826950073 0.00015699760597511416
hpackheaders_canonical_form_trimming_whitespace 0.04422605037689209 0.044754981994628906 0.04440640211105347 0.00023535089672761531
hpackheaders_canonical_form_trimming_whitespace_short_strings 0.04088902473449707 0.04139399528503418 0.04100780487060547 0.00019896661103882608
hpackheaders_canonical_form_trimming_whitespace_long_strings 0.04970097541809082 0.05019199848175049 0.04985220432281494 0.00021980457652302524
huffman_encode_basic 0.7676070928573608 0.7695299386978149 0.7681055784225463 0.0005364138489963227
huffman_encode_complex 0.5194940567016602 0.5202770233154297 0.5198963046073913 0.0002579057159458666
huffman_decode_basic 0.056532979011535645 0.05703103542327881 0.056725692749023435 0.0002025397769603034
huffman_decode_complex 0.04661297798156738 0.04776501655578613 0.046978187561035153 0.0004299907412052386
server_only_10k_requests_1_concurrent 0.3618440628051758 0.36552298069000244 0.3626688003540039 0.001039615633965984
server_only_10k_requests_100_concurrent 0.2846330404281616 0.28559303283691406 0.2850819110870361 0.00035980189089376255
stream_teardown_10k_requests_100_concurrent 0.15219998359680176 0.15268898010253906 0.152490496635437 0.00019064078063663374

comparison

glbrntt commented 2 years ago

@swift-nio-bot test perf please

swift-server-bot commented 2 years ago

performance report

build id: 15

timestamp: Wed Sep 29 12:55:11 UTC 2021

results

nameminmaxmeanstd
1_conn_10k_reqs 1.6213979721069336 1.6442910432815552 1.632042407989502 0.006699844032143084
encode_100k_header_blocks_indexable 0.14471304416656494 0.14747297763824463 0.14572598934173583 0.0008046368602636831
encode_100k_header_blocks_nonindexable 0.3090909719467163 0.3160330057144165 0.3103432893753052 0.00211925033167702
encode_100k_header_blocks_neverIndexed 0.31516897678375244 0.31856203079223633 0.31606420278549197 0.0011066502001272586
decode_100k_header_blocks_indexable 0.11055600643157959 0.11164200305938721 0.11097080707550049 0.0003699525350451268
decode_100k_header_blocks_nonindexable 0.13858795166015625 0.13942408561706543 0.13902969360351564 0.0002693567257094209
decode_100k_header_blocks_neverIndexed 0.13593900203704834 0.13771402835845947 0.13646239042282104 0.00048347271333613114
hpackheaders_canonical_form 0.031477928161621094 0.032000064849853516 0.03155630826950073 0.00015699760597511416
hpackheaders_canonical_form_trimming_whitespace 0.04422605037689209 0.044754981994628906 0.04440640211105347 0.00023535089672761531
hpackheaders_canonical_form_trimming_whitespace_short_strings 0.04088902473449707 0.04139399528503418 0.04100780487060547 0.00019896661103882608
hpackheaders_canonical_form_trimming_whitespace_long_strings 0.04970097541809082 0.05019199848175049 0.04985220432281494 0.00021980457652302524
huffman_encode_basic 0.7676070928573608 0.7695299386978149 0.7681055784225463 0.0005364138489963227
huffman_encode_complex 0.5194940567016602 0.5202770233154297 0.5198963046073913 0.0002579057159458666
huffman_decode_basic 0.056532979011535645 0.05703103542327881 0.056725692749023435 0.0002025397769603034
huffman_decode_complex 0.04661297798156738 0.04776501655578613 0.046978187561035153 0.0004299907412052386
server_only_10k_requests_1_concurrent 0.3618440628051758 0.36552298069000244 0.3626688003540039 0.001039615633965984
server_only_10k_requests_100_concurrent 0.2846330404281616 0.28559303283691406 0.2850819110870361 0.00035980189089376255
stream_teardown_10k_requests_100_concurrent 0.15219998359680176 0.15268898010253906 0.152490496635437 0.00019064078063663374

comparison

glbrntt commented 2 years ago

@yim-lee looks like there's a credentials problem with the perf tests, could you take a look?

yim-lee commented 2 years ago

@swift-nio-bot test perf please

swift-server-bot commented 2 years ago

performance report

build id: 19

timestamp: Mon May 16 21:11:03 UTC 2022

results

nameminmaxmeanstd
1_conn_10k_reqs 1.6620510816574097 1.7093840837478638 1.6752350091934205 0.013837931271847511
encode_100k_header_blocks_indexable 0.14597594738006592 0.14680397510528564 0.1464395046234131 0.00025325488026855907
encode_100k_header_blocks_nonindexable 0.3263270854949951 0.32759296894073486 0.3268532037734985 0.00033949887211778964
encode_100k_header_blocks_neverIndexed 0.33021700382232666 0.3313760757446289 0.3308082938194275 0.0003574828562520098
decode_100k_header_blocks_indexable 0.11215996742248535 0.11271500587463379 0.11250079870223999 0.0002270040612611711
decode_100k_header_blocks_nonindexable 0.1363849639892578 0.1368640661239624 0.13667590618133546 0.00017045351102189813
decode_100k_header_blocks_neverIndexed 0.13660502433776855 0.13744497299194336 0.1370334029197693 0.0002582564106823645
hpackheaders_canonical_form 0.030714035034179688 0.0328899621963501 0.031182992458343505 0.00077543752004855
hpackheaders_canonical_form_trimming_whitespace 0.04084599018096924 0.041416049003601074 0.04101650714874268 0.00021280351643373626
hpackheaders_canonical_form_trimming_whitespace_short_strings 0.03820502758026123 0.038758039474487305 0.038365304470062256 0.000211206240084058
hpackheaders_canonical_form_trimming_whitespace_long_strings 0.0451129674911499 0.04567098617553711 0.04530678987503052 0.00023332187725221852
hpackheaders_normalize_httpheaders_removing_10k_conn_headers 7.82142698764801 7.856073021888733 7.83308219909668 0.009451634380621667
hpackheaders_normalize_httpheaders_keeping_10k_conn_headers 15.313019037246704 15.415313005447388 15.377971398830415 0.0268216753656464
huffman_encode_basic 0.7833390235900879 0.785862922668457 0.7844860911369324 0.000814594877539253
huffman_encode_complex 0.5246419906616211 0.5258079767227173 0.5250432968139649 0.00034582772282528617
huffman_decode_basic 0.05712294578552246 0.05763602256774902 0.057291197776794436 0.0002022262318410239
huffman_decode_complex 0.04519498348236084 0.045616984367370605 0.04531759023666382 0.00015196663672409472
server_only_10k_requests_1_concurrent 0.357558012008667 0.3583940267562866 0.35797479152679446 0.0002200199037565395
server_only_10k_requests_100_concurrent 0.292153000831604 0.2925199270248413 0.2923703908920288 0.00012485392653978868
stream_teardown_10k_requests_100_concurrent 0.15442097187042236 0.15515899658203125 0.1548266053199768 0.0002299860392892099

comparison

name current previous winner diff
1_conn_10k_reqs 1.6752350091934205 1.9301332116127015 current -13%
encode_100k_header_blocks_indexable 0.1464395046234131 0.1595517873764038 current -8%
encode_100k_header_blocks_nonindexable 0.3268532037734985 0.29965169429779054 previous 9%
encode_100k_header_blocks_neverIndexed 0.3308082938194275 0.300789999961853 previous 10%
decode_100k_header_blocks_indexable 0.11250079870223999 0.19248170852661134 current -41%
decode_100k_header_blocks_nonindexable 0.13667590618133546 0.2619980931282043 current -47%
decode_100k_header_blocks_neverIndexed 0.1370334029197693 0.261506187915802 current -47%
hpackheaders_canonical_form 0.031182992458343505 n/a n/a%
hpackheaders_canonical_form_trimming_whitespace 0.04101650714874268 n/a n/a%
hpackheaders_canonical_form_trimming_whitespace_short_strings 0.038365304470062256 n/a n/a%
hpackheaders_canonical_form_trimming_whitespace_long_strings 0.04530678987503052 n/a n/a%
hpackheaders_normalize_httpheaders_removing_10k_conn_headers 7.83308219909668 n/a n/a%
hpackheaders_normalize_httpheaders_keeping_10k_conn_headers 15.377971398830415 n/a n/a%
huffman_encode_basic 0.7844860911369324 0.30517669916152956 previous 157%
huffman_encode_complex 0.5250432968139649 0.25399419069290163 previous 106%
huffman_decode_basic 0.057291197776794436 0.0629040002822876 current -8%
huffman_decode_complex 0.04531759023666382 0.04881941080093384 current -7%
server_only_10k_requests_1_concurrent 0.35797479152679446 0.47175098657608033 current -24%
server_only_10k_requests_100_concurrent 0.2923703908920288 0.3798054099082947 current -23%
stream_teardown_10k_requests_100_concurrent 0.1548266053199768 0.20033220052719117 current -22%

significant differences found

yim-lee commented 2 years ago

Fixed performance jobs and updated them to use Swift 5.6.

glbrntt commented 2 years ago

Fixed performance jobs and updated them to use Swift 5.6.

Thanks Yim!

glbrntt commented 2 years ago

@swift-nio-bot test perf please

swift-server-bot commented 2 years ago

performance report

build id: 20

timestamp: Tue May 17 08:03:15 UTC 2022

results

nameminmaxmeanstd
1_conn_10k_reqs 1.6446770429611206 1.6588588953018188 1.6514891862869263 0.0048384351602989715
encode_100k_header_blocks_indexable 0.14113497734069824 0.14191603660583496 0.14149559736251832 0.00020879182882760055
encode_100k_header_blocks_nonindexable 0.32961010932922363 0.33041298389434814 0.3300792098045349 0.0002178042235342109
encode_100k_header_blocks_neverIndexed 0.3290550708770752 0.3306189775466919 0.33014119863510133 0.0004438579365258506
decode_100k_header_blocks_indexable 0.10575008392333984 0.10828197002410889 0.10633149147033691 0.0007290222046684748
decode_100k_header_blocks_nonindexable 0.12971198558807373 0.13060498237609863 0.13021708726882936 0.00027991397448224345
decode_100k_header_blocks_neverIndexed 0.129256010055542 0.13002800941467285 0.1296894907951355 0.00025542470193778424
hpackheaders_canonical_form 0.031186938285827637 0.031751036643981934 0.03133831024169922 0.00021196263010964168
hpackheaders_canonical_form_trimming_whitespace 0.042424917221069336 0.04302096366882324 0.04259538650512695 0.00021009214520661162
hpackheaders_canonical_form_trimming_whitespace_short_strings 0.03938698768615723 0.039901018142700195 0.03958319425582886 0.00018086169031722804
hpackheaders_canonical_form_trimming_whitespace_long_strings 0.048689961433410645 0.050512075424194336 0.049069106578826904 0.0005619627883776512
hpackheaders_normalize_httpheaders_removing_10k_conn_headers 7.736546993255615 7.763311982154846 7.741343700885773 0.008176557301024788
hpackheaders_normalize_httpheaders_keeping_10k_conn_headers 14.957084059715271 15.066746950149536 14.979897391796111 0.03228361539152468
huffman_encode_basic 0.7551519870758057 0.7559449672698975 0.7555625081062317 0.0003255647143439214
huffman_encode_complex 0.49998903274536133 0.5005970001220703 0.5002685785293579 0.00019643443559291295
huffman_decode_basic 0.05694591999053955 0.05811595916748047 0.05721179246902466 0.00035247995794014626
huffman_decode_complex 0.042752981185913086 0.043218016624450684 0.042886900901794436 0.0001692367055323578
server_only_10k_requests_1_concurrent 0.3547559976577759 0.35523998737335205 0.35498679876327516 0.00019125718258984167
server_only_10k_requests_100_concurrent 0.2849010229110718 0.2857990264892578 0.2853196024894714 0.0002804569409385593
stream_teardown_10k_requests_100_concurrent 0.14845693111419678 0.14928603172302246 0.14884350299835206 0.00022794830466246426

comparison

name current previous winner diff
1_conn_10k_reqs 1.6514891862869263 1.9301332116127015 current -14%
encode_100k_header_blocks_indexable 0.14149559736251832 0.1595517873764038 current -11%
encode_100k_header_blocks_nonindexable 0.3300792098045349 0.29965169429779054 previous 10%
encode_100k_header_blocks_neverIndexed 0.33014119863510133 0.300789999961853 previous 9%
decode_100k_header_blocks_indexable 0.10633149147033691 0.19248170852661134 current -44%
decode_100k_header_blocks_nonindexable 0.13021708726882936 0.2619980931282043 current -50%
decode_100k_header_blocks_neverIndexed 0.1296894907951355 0.261506187915802 current -50%
hpackheaders_canonical_form 0.03133831024169922 n/a n/a%
hpackheaders_canonical_form_trimming_whitespace 0.04259538650512695 n/a n/a%
hpackheaders_canonical_form_trimming_whitespace_short_strings 0.03958319425582886 n/a n/a%
hpackheaders_canonical_form_trimming_whitespace_long_strings 0.049069106578826904 n/a n/a%
hpackheaders_normalize_httpheaders_removing_10k_conn_headers 7.741343700885773 n/a n/a%
hpackheaders_normalize_httpheaders_keeping_10k_conn_headers 14.979897391796111 n/a n/a%
huffman_encode_basic 0.7555625081062317 0.30517669916152956 previous 147%
huffman_encode_complex 0.5002685785293579 0.25399419069290163 previous 97%
huffman_decode_basic 0.05721179246902466 0.0629040002822876 current -9%
huffman_decode_complex 0.042886900901794436 0.04881941080093384 current -12%
server_only_10k_requests_1_concurrent 0.35498679876327516 0.47175098657608033 current -24%
server_only_10k_requests_100_concurrent 0.2853196024894714 0.3798054099082947 current -24%
stream_teardown_10k_requests_100_concurrent 0.14884350299835206 0.20033220052719117 current -25%

significant differences found