HTTP/2 functional tests

[krizhanovsky]

The main part is that (also fix the disabled tests for the issue)

• [X] deproxy client must be extended to support HTTP/2, just like it was done for TLS functional tests. Note that curl supports HTTP/2 using nfhttp2 library, so probably it'd be more convenient to implement some HTTP/2 tests using curl or the library directly. Probably it makes sense to fork and extend h2spec rather than develop h2 layer in python in our own. DONE: Deproxy was extended to support http2 by commit https://github.com/tempesta-tech/tempesta-test/commit/6e521d3ba6a04f42e4b71b3aa863c8014ea34d20

At least following new functional tests must be implemented for HTTP/2 (from more crucial to lower priority). It seems some of the tests are covered by h2spec and some of them are covered by already existing h2 tests:

• [X] pipelining test for https://github.com/tempesta-tech/tempesta/issues/1619 , e.g. extending pipelining/test_pipelining.py with the scenario for HTTP/2

• [x] Check that Tempesta generates correct static responses (e.g. 304)

• [x] Redirect (e.g. for JavaScript challenge) must be built with correct HTTP2 framing

• [x] Sticky cookie via h2 protocol, when multiple cookies are send by client (https://github.com/tempesta-tech/tempesta/issues/1396)

• [x] Test POST request with a body (https://github.com/tempesta-tech/tempesta/issues/1409)

• [x] A client sends data by 1-byte chunks to stress HTTP framing and parsing

• [x] Test that the same header (e.g. Host) is normally processed if different HPACK methods are used (only static index, only Huffman encoding, no HPACK at all (plain text)).

• [x] HPACK bomb

• [x] ALPN - establish h2 and http/1.1 connections and make sure that the higher priority protocol is used if both of them are announced by a client and configured on the server.

• [x] Verification of correct frames exchange during establishing of h2 connection: get correct SETTINGS frame (with SETTINGS_INITIAL_WINDOW_SIZE parameter), correct WINDOW_UPDATE frame and correct SETTINGS ACK frame.

• [x] Get response on PING frame.

• [x] ~Basic HTTP/2 request, convert it to HTTP/1.1 request to a backend, and forward the response in HTTP/2 format.~ Co ered by any basic HTTP/2 test case

• [x] Verification of correct handling for connection and stream error cases: a) get valid GOAWAY frame in response on connection error case; b) get valid RST_STREAM frame in response on stream error situation (the whole connection must remain alive).

• [x] ~Check correct stream processing in HTTP2_STREAM_[LOC|REM]_CLOSED state: 1. Instigate the stream error case for particular stream; 2. Send predefined amount of frames (service and application) for that stream - frames must be silently ignored by TempestaFW (i.e. frames must not be passed to backend, and errors must not be generated); 3. After the predefined amount of frames will be exceeded, TempestaFW must return connection error to client peer (GOAWAY frame) and close connection.~ h2spec: http2/5.1

• [x] Basic test for streams multiplexing: 1. Different HTTP/2 requests (i.e. HEADERS/CONTINUATION/DATA frames in each request) are sent in several streams - one by one, without waiting the responses; 2. And correct HTTP/2 responses must be received for each request/stream (regardless of the sending/receiving order). Do this in one and several TCP connections in parallel.

• [x] Exceed number of allowed streams and check that RST is received

• [x] HPACK entries eviction: we must correctly process the maximum dynamic table and evict old entries - the test must define the table limit and send too many different headers

• [x] ~Emulate dynamic table size tracking bug on any of the peers: encoder thinks (incorrectly) that there is enough memory and add a new dynamic table entry sending in a request both the new and the oldest (evicted on Tempesta side) entries. Tempesta must correctly handle the encoding error.~ h2spec: http2/4.3 and hpack/2.3.3

• [x] Performance/stress test with many concurrent TCP connections with many concurrent HTTP/2 streams. Use h2load from https://github.com/nghttp2/nghttp2 .

• [x] ~Fuzzing for various frame formats - we need to intensively test handling of invalid frames on different stream stage, not only initial frames.~ We have "some" HTTP/2 fuzzer in the functional tests https://github.com/tempesta-tech/tempesta/pull/1603

• [x] Use small SETTINGS_MAX_FRAME_SIZE on a client and test that HTTP response headers and body are normally fragmented

• [x] Test case from https://github.com/tempesta-tech/tempesta/issues/1617 (also see discussion https://github.com/tempesta-tech/tempesta/pull/1635#discussion_r896269228) send a header with index 2 (static table for :method GET) AND GET as string value, i.e. use the full index in a partially indexed pseudoheader.

• [x] ~Check correct processing of padded frames (form DATA/HEADERS frames with padding).~ h2spec: generic3.1 and 3.2

• [x] Stream IDs: (1) check that even IDs and ID > 0x7fffffff aren't accepted and (2) you cannot reuse old ID (see Victim 1–HTTP/2 Stream Multiplexing (CVE-2016-0150))

• [x] Check that only allowed frames are successfully received on closed stream RFC 7540 5.1 closed paragraph.

Following existing HTTP/1 tests must be extended, i.e. each test must run twice in HTTP/1 and HTTP/2 modes correspondingly:

• [x] ~/cache - need to check that HTTP/1.1 response is correctly sent to HTTP/1.1 clients and transformed to HTTP/2 for HTTP/2 client.~ #810
• [x] /http_rules
• [x] ~/malformed~ #427
• [x] ~/mixed_requests~
• [x] /sched
• [x] /frang
• [x] /leaks (many iterations of streams freeing must be performed)

[vankoven]

The h2spec tool provides a lot of tests for http2 connections and must be also used. It provides tests:

Generic tests for HTTP/2 server
  1. Starting HTTP/2
    1: Sends a client connection preface

  2. Streams and Multiplexing
    1: Sends a PRIORITY frame on idle stream
    2: Sends a WINDOW_UPDATE frame on half-closed (remote) stream
    3: Sends a PRIORITY frame on half-closed (remote) stream
    4: Sends a RST_STREAM frame on half-closed (remote) stream
    5: Sends a PRIORITY frame on closed stream

  3. Frame Definitions
    3.1. DATA
      1: Sends a DATA frame
      2: Sends multiple DATA frames
      3: Sends a DATA frame with padding

    3.2. HEADERS
      1: Sends a HEADERS frame
      2: Sends a HEADERS frame with padding
      3: Sends a HEADERS frame with priority

    3.3. PRIORITY
      1: Sends a PRIORITY frame with priority 1
      2: Sends a PRIORITY frame with priority 256
      3: Sends a PRIORITY frame with stream dependency
      4: Sends a PRIORITY frame with exclusive
      5: Sends a PRIORITY frame for an idle stream, then send a HEADER frame for a lower stream ID

    3.4. RST_STREAM
      1: Sends a RST_STREAM frame

    3.5. SETTINGS
      1: Sends a SETTINGS frame

    3.7. PING
      1: Sends a PING frame

    3.8. GOAWAY
      1: Sends a GOAWAY frame

    3.9. WINDOW_UPDATE
      1: Sends a WINDOW_UPDATE frame with stream ID 0
      2: Sends a WINDOW_UPDATE frame with stream ID 1

    3.10. CONTINUATION
      1: Sends a CONTINUATION frame
      2: Sends multiple CONTINUATION frames

  4. HTTP Message Exchanges
    1: Sends a GET request
    2: Sends a HEAD request
    3: Sends a POST request
    4: Sends a POST request with trailers

  5. HPACK
    1: Sends a indexed header field representation
    2: Sends a literal header field with incremental indexing - indexed name
    3: Sends a literal header field with incremental indexing - indexed name (with Huffman coding)
    4: Sends a literal header field with incremental indexing - new name
    5: Sends a literal header field with incremental indexing - new name (with Huffman coding)
    6: Sends a literal header field without indexing - indexed name
    7: Sends a literal header field without indexing - indexed name (with Huffman coding)
    8: Sends a literal header field without indexing - new name
    9: Sends a literal header field without indexing - new name (huffman encoded)
    10: Sends a literal header field never indexed - indexed name
    11: Sends a literal header field never indexed - indexed name (huffman encoded)
    12: Sends a literal header field never indexed - new name
    13: Sends a literal header field never indexed - new name (huffman encoded)
    14: Sends a dynamic table size update
    15: Sends multiple dynamic table size update

Hypertext Transfer Protocol Version 2 (HTTP/2)
  3. Starting HTTP/2
    3.5. HTTP/2 Connection Preface
      1: Sends client connection preface
      2: Sends invalid connection preface

  4. HTTP Frames
    4.1. Frame Format
      1: Sends a frame with unknown type
      2: Sends a frame with undefined flag
      3: Sends a frame with reserved field bit

    4.2. Frame Size
      1: Sends a DATA frame with 2^14 octets in length
      2: Sends a large size DATA frame that exceeds the SETTINGS_MAX_FRAME_SIZE
      3: Sends a large size HEADERS frame that exceeds the SETTINGS_MAX_FRAME_SIZE

    4.3. Header Compression and Decompression
      1: Sends invalid header block fragment
      2: Sends a PRIORITY frame while sending the header blocks
      3: Sends a HEADERS frame to another stream while sending the header blocks

  5. Streams and Multiplexing
    5.1. Stream States
      1: idle: Sends a DATA frame
      2: idle: Sends a RST_STREAM frame
      3: idle: Sends a WINDOW_UPDATE frame
      4: idle: Sends a CONTINUATION frame
      5: half closed (remote): Sends a DATA frame
      6: half closed (remote): Sends a HEADERS frame
      7: half closed (remote): Sends a CONTINUATION frame
      8: closed: Sends a DATA frame after sending RST_STREAM frame
      9: closed: Sends a HEADERS frame after sending RST_STREAM frame
      10: closed: Sends a CONTINUATION frame after sending RST_STREAM frame
      11: closed: Sends a DATA frame
      12: closed: Sends a HEADERS frame
      13: closed: Sends a CONTINUATION frame

      5.1.1. Stream Identifiers
        1: Sends even-numbered stream identifier
        2: Sends stream identifier that is numerically smaller than previous

      5.1.2. Stream Concurrency
        1: Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded

    5.3. Stream Priority
      5.3.1. Stream Dependencies
        1: Sends HEADERS frame that depend on itself
        2: Sends PRIORITY frame that depend on itself

    5.4. Error Handling
      5.4.1. Connection Error Handling
        1: Sends an invalid PING frame for connection close

    5.5. Extending HTTP/2
      1: Sends an unknown extension frame
      2: Sends an unknown extension frame in the middle of a header block

  6. Frame Definitions
    6.1. DATA
      1: Sends a DATA frame with 0x0 stream identifier
      2: Sends a DATA frame on the stream that is not in "open" or "half-closed (local)" state
      3: Sends a DATA frame with invalid pad length

    6.2. HEADERS
      1: Sends a HEADERS frame without the END_HEADERS flag, and a PRIORITY frame
      2: Sends a HEADERS frame to another stream while sending a HEADERS frame
      3: Sends a HEADERS frame with 0x0 stream identifier
      4: Sends a HEADERS frame with invalid pad length

    6.3. PRIORITY
      1: Sends a PRIORITY frame with 0x0 stream identifier
      2: Sends a PRIORITY frame with a length other than 5 octets

    6.4. RST_STREAM
      1: Sends a RST_STREAM frame with 0x0 stream identifier
      2: Sends a RST_STREAM frame on a idle stream
      3: Sends a RST_STREAM frame with a length other than 4 octets

    6.5. SETTINGS
      1: Sends a SETTINGS frame with ACK flag and payload
      2: Sends a SETTINGS frame with a stream identifier other than 0x0
      3: Sends a SETTINGS frame with a length other than a multiple of 6 octets

      6.5.2. Defined SETTINGS Parameters
        1: SETTINGS_ENABLE_PUSH (0x2): Sends the value other than 0 or 1
        2: SETTINGS_INITIAL_WINDOW_SIZE (0x4): Sends the value above the maximum flow control window size
        3: SETTINGS_MAX_FRAME_SIZE (0x5): Sends the value below the initial value
        4: SETTINGS_MAX_FRAME_SIZE (0x5): Sends the value above the maximum allowed frame size
        5: Sends a SETTINGS frame with unknown identifier

      6.5.3. Settings Synchronization
        1: Sends multiple values of SETTINGS_INITIAL_WINDOW_SIZE
        2: Sends a SETTINGS frame without ACK flag

    6.7. PING
      1: Sends a PING frame
      2: Sends a PING frame with ACK
      3: Sends a PING frame with a stream identifier field value other than 0x0
      4: Sends a PING frame with a length field value other than 8

    6.8. GOAWAY
      1: Sends a GOAWAY frame with a stream identifier other than 0x0

    6.9. WINDOW_UPDATE
      1: Sends a WINDOW_UPDATE frame with a flow control window increment of 0
      2: Sends a WINDOW_UPDATE frame with a flow control window increment of 0 on a stream
      3: Sends a WINDOW_UPDATE frame with a length other than 4 octets

      6.9.1. The Flow-Control Window
        1: Sends SETTINGS frame to set the initial window size to 1 and sends HEADERS frame
        2: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1
        3: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream

      6.9.2. Initial Flow-Control Window Size
        1: Changes SETTINGS_INITIAL_WINDOW_SIZE after sending HEADERS frame
        2: Sends a SETTINGS frame for window size to be negative
        3: Sends a SETTINGS_INITIAL_WINDOW_SIZE settings with an exceeded maximum window size value

    6.10. CONTINUATION
      1: Sends multiple CONTINUATION frames preceded by a HEADERS frame
      2: Sends a CONTINUATION frame followed by any frame other than CONTINUATION
      3: Sends a CONTINUATION frame with 0x0 stream identifier
      4: Sends a CONTINUATION frame preceded by a HEADERS frame with END_HEADERS flag
      5: Sends a CONTINUATION frame preceded by a CONTINUATION frame with END_HEADERS flag
      6: Sends a CONTINUATION frame preceded by a DATA frame

  7. Error Codes
    1: Sends a GOAWAY frame with unknown error code
    2: Sends a RST_STREAM frame with unknown error code

  8. HTTP Message Exchanges
    8.1. HTTP Request/Response Exchange
      1: Sends a second HEADERS frame without the END_STREAM flag

      8.1.2. HTTP Header Fields
        1: Sends a HEADERS frame that contains the header field name in uppercase letters

        8.1.2.1. Pseudo-Header Fields
          1: Sends a HEADERS frame that contains a unknown pseudo-header field
          2: Sends a HEADERS frame that contains the pseudo-header field defined for response
          3: Sends a HEADERS frame that contains a pseudo-header field as trailers
          4: Sends a HEADERS frame that contains a pseudo-header field that appears in a header block after a regular header field

        8.1.2.2. Connection-Specific Header Fields
          1: Sends a HEADERS frame that contains the connection-specific header field
          2: Sends a HEADERS frame that contains the TE header field with any value other than "trailers"

        8.1.2.3. Request Pseudo-Header Fields
          1: Sends a HEADERS frame with empty ":path" pseudo-header field
          2: Sends a HEADERS frame that omits ":method" pseudo-header field
          3: Sends a HEADERS frame that omits ":scheme" pseudo-header field
          4: Sends a HEADERS frame that omits ":path" pseudo-header field
          5: Sends a HEADERS frame with duplicated ":method" pseudo-header field
          6: Sends a HEADERS frame with duplicated ":scheme" pseudo-header field
          7: Sends a HEADERS frame with duplicated ":path" pseudo-header field

        8.1.2.6. Malformed Requests and Responses
          1: Sends a HEADERS frame with the "content-length" header field which does not equal the DATA frame payload length
          2: Sends a HEADERS frame with the "content-length" header field which does not equal the sum of the multiple DATA frames payload length

    8.2. Server Push
      1: Sends a PUSH_PROMISE frame

HPACK: Header Compression for HTTP/2
  2. Compression Process Overview
    2.3. Indexing Tables
      2.3.3. Index Address Space
        1: Sends a header field representation with invalid index

  4. Dynamic Table Management
    4.2. Maximum Table Size
      1: Sends a dynamic table size update at the end of header block

  5. Primitive Type Representations
    5.2. String Literal Representation
      1: Sends a Huffman-encoded string literal representation with padding longer than 7 bits
      2: Sends a Huffman-encoded string literal representation padded by zero
      3: Sends a Huffman-encoded string literal representation containing the EOS symbol

  6. Binary Format
    6.1. Indexed Header Field Representation
      1: Sends a indexed header field representation with index 0

    6.3. Dynamic Table Size Update
      1: Sends a dynamic table size update larger than the value of SETTINGS_HEADER_TABLE_SIZE

Utility is easy started: ./h2spec -h <tempesta_ip> -p 443 -t -k (-t for TLS and -k to skip server certificate checks). Tempesta must be configured to serve / requests with 200 status code. The h2spec returns overall status in return code, 0 for ok, 1 if at least some tests has failed, so it can be easily integrated in our CI. We can also specify which test cases to run, but excluding some tests is not possible

[vankoven]

Tests for tempesta-tech/tempesta#1378 need folowing scenarios implemented:

• [x] response without body and with short headers
• [x] response with long set of headers and no body
• [x] response with short set of headers and short body, body must be shorter than one skb fragment (one page)
• [x] response with short set of headers and long body, body must be big enough to occupy more than one skb
• [x] response with long set of headers and long body, body must be big enough to occupy more than one skb

For all scenarios above:

• [x] response is forwarded from origin
• [x] response is served from cache
• [x] client is connected via http/2 protocol and receives correctly framed response
• [x] client is connected via http/1 protocol (both http and https) and receives response without h2 frame information
• [x] when served via encrypted connections, must receive multiple responses to check that no data corruption happen due to inplace crypto operations
• [x] check that the received payload is always the same whatever transport protocol is used

In all the cases read short as 'less than maximum allowed frame size', and long as 'bigger than allowed frame size'. Need to run tests with various allowed frame sizes: default frame size (16kb) and user settings (any value bigger than 16kb)

[krizhanovsky]

Just faced HTTPX Python client for HTTP/2, see also https://pypi.org/project/httpx/ . Probably. it's the most easy way to test HTTP/2. Requires Python 3 though, so it will depend on https://github.com/tempesta-tech/tempesta-test/issues/56

[avbelov23]

May be help python-hyper/h2

[krizhanovsky]

https://github.com/tempesta-tech/tempesta-test/pull/164 implements tests for chunked modifications

[KonsKo]

h2-frang-related tests (moved from https://github.com/tempesta-tech/tempesta/issues/673).

• [x] Check that Tempesta sends SETTINGS_MAX_HEADER_LIST_SIZE settings parameter to a client if http_header_cnt HTTP limit is enabled.
• [x] Check that client_header_timeout limits time from connection established and long TLS handhshake is correctlyu blocked with the limit. If so, please update description of the limit at the HTTP limits Wiki
• [x] Cause HPACK bomb, probably with many connections if necessary, and make sure that http_header_cnt limit prevents the attack.
• [x] Cause response timeout for HTTP/2 requests
• [x] Check that all the current Frang limits work with HTTP/2
• [x] ~HTTP/2 streams rate limit to prevent HTTP/2 slow read (#1346).~ moved to #1346
• [x] ~HTTP/2 frames rate limit to prevent HTTP/2 floods (#1346).~ moved to #1346
• [x] ~Drop HTTP session on security events (#1045)~ moved to #1045

[krizhanovsky]

• [x] Please also make an HTTP/2 IPv6 test, which at least load some data

[const-t]

• [x] ~Check that headers which were added to response using resp_hdr_set are correctly hpack encoded using hpack static/dynamic table.~ #409

[RomanBelozerov]

Closed as completed