supported HTTP/2

[chobits]

try to fix: https://github.com/chobits/ngx_http_proxy_connect_module/issues/25

RFC: https://httpwg.org/specs/rfc7540.html#CONNECT

WIP: not ready now

[chobits]

phrase from RFC as following

1. Client sends HEADERS frame (:method = CONNECT, :authority: <host>[:<port>]) to the proxy server.
2. The proxy server establishes a connection with backend server, then it sends HEADERS frame with 2xx status.
3. The proxy server reads data from backend/client, then transmits the received data to client/backend.
   • if proxy server reads data from backend, then assemble it into DATA frame and sent it to client.
   • if proxy server reads DATA frame from client, then send the payload of DATA frame to backend .

[chobits]

HINT: How upstream reads request body from http/2 DATA frame:

If we dont wanna modify the http/2 logic in the nginx core, we need to use the logic of upstream reading request body.

ngx_http_v2_state_read_data
 -> ngx_http_v2_process_request_body(r, pos) // pos --> h2c head buf
   ngx_memcopy pos -> r->request_body->buf->last
                 data saved in r->request->body->buf->[pos, last]
   -> ngx_http_v2_filter_request_body(r)
         cl->b <= r->request->body->buf
         r->request_body->busy/free <-- ngx_chain_update_chains --- cl->b
   -> rb->post_handler(r);
       -> ngx_http_upstream_init
          -> ngx_http_upstream_init_request

              if (r->request_body) {
                  u->request_bufs = r->request_body->bufs;
              }

            -> ngx_http_upstream_connect
             -> ngx_http_upstream_send_request
              -> ngx_http_upstream_send_request_body(r, u, do_write);
                out <- u->request_bufs
                r->request_body->bufs <- NULL
                for ( ;; ) {
                     if (do_write) {
                         rc = ngx_output_chain(&u->output, out);    // send to upstream fd
                     }
                     ...
                }

[chobits]

source notes for http/2 request body reading logic( DATA frame receiving logic):

------
http/2 connection handler：
    rev->handler = ngx_http_v2_read_handler;
                      -> ngx_http_v2_state_head
                        -> ngx_http_v2_state_data
    c->write->handler = ngx_http_v2_write_handler;

------
ngx_http_v2_state_read_data
 -> ngx_http_v2_process_request_body(r, pos) // pos --> h2c head buf
   -> ngx_memcopy(pos -> r->request_body->buf->last)
      // data saved in r->request->body->buf->[pos, last]
   -> ngx_http_v2_filter_request_body(r)
         cl->b <= r->request->body->buf
         r->request_body->busy/free <-- ngx_chain_update_chains --- cl->b
   -> rb->post_handler(r);              // TODO: is it ngx_http_init_upstream always?
       -> ngx_http_upstream_init
          -> ngx_http_upstream_init_request

              if (r->request_body) {
                  u->request_bufs = r->request_body->bufs;
              }

            -> ngx_http_upstream_connect
             -> ngx_http_upstream_send_request
              -> ngx_http_upstream_send_request_body(r, u, do_write);
                out <- u->request_bufs
                r->request_body->bufs <- NULL
                for ( ;; ) {
                    if (do_write) {
                        rc = ngx_output_chain(&u->output, out);    // send to upstream fd
                    }

                    if (r->reading_body) {
                        /* read client request body */
                        rc = ngx_http_read_unbuffered_request_body(r);
                              -> ngx_http_v2_read_unbuffered_request_body(r);
                                -> ngx_http_v2_process_request_body
                                  -> ngx_http_v2_filter_request_body
                    }
                }

[chobits]

If proxy_connect does not connect to backend server and DATA frame from client has already been sent to nginx, the logic backtrace is as following:

1. header frame + data frame ==> nginx
2. handle header frame , connecting to backend but waiting establishment of this connection
3. handle DATA frame ---. following bt log

(gdb)
1190        return ngx_http_v2_state_complete(h2c, pos, end);
(gdb) bt
#0  ngx_http_v2_state_read_data (h2c=h2c@entry=0x562d049be0d0, pos=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n", pos@entry=0x562d049d18f9 "GET /hello HTTP/1.1\r\nHost: test.com\r\n\r\nhello HTTP/1.1\r\nHost: test.com\r\n\r\n", end=end@entry=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n") at src/http/v2/ngx_http_v2.c:1190
#1  0x0000562d038190c7 in ngx_http_v2_state_data (h2c=0x562d049be0d0, pos=0x562d049d18f9 "GET /hello HTTP/1.1\r\nHost: test.com\r\n\r\nhello HTTP/1.1\r\nHost: test.com\r\n\r\n", end=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n") at src/http/v2/ngx_http_v2.c:1087
#2  0x0000562d038175c6 in ngx_http_v2_state_head (h2c=0x562d049be0d0, pos=0x562d049d18f9 "GET /hello HTTP/1.1\r\nHost: test.com\r\n\r\nhello HTTP/1.1\r\nHost: test.com\r\n\r\n", end=0x562d049d1920 "hello HTTP/1.1\r\nHost: test.com\r\n\r\n") at src/http/v2/ngx_http_v2.c:947
#3  0x0000562d038196cb in ngx_http_v2_read_handler (rev=0x7f98954e4250) at src/http/v2/ngx_http_v2.c:435
#4  0x0000562d037c2221 in ngx_epoll_process_events (cycle=0x562d048a4c80, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901
#5  0x0000562d037b654e in ngx_process_events_and_timers (cycle=cycle@entry=0x562d048a4c80) at src/event/ngx_event.c:248
#6  0x0000562d037bfd8f in ngx_worker_process_cycle (cycle=0x562d048a4c80, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:721
#7  0x0000562d037be1c3 in ngx_spawn_process (cycle=cycle@entry=0x562d048a4c80, proc=proc@entry=0x562d037bfc52 <ngx_worker_process_cycle>, data=data@entry=0x0, name=name@entry=0x562d038c96ea "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:199
#8  0x0000562d037bf4c1 in ngx_start_worker_processes (cycle=cycle@entry=0x562d048a4c80, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:344
#9  0x0000562d037c06b0 in ngx_master_process_cycle (cycle=0x562d048a4c80) at src/os/unix/ngx_process_cycle.c:130
#10 0x0000562d037942ed in main (argc=1, argv=<optimized out>) at src/core/nginx.c:383

• debug log

2023/07/31 15:14:22 [debug] 522#0: *16 event timer add: 17: 10000:22052976
2023/07/31 15:14:22 [debug] 522#0: *16 http finalize request: -4, "?" a:1, c:2
2023/07/31 15:14:22 [debug] 522#0: *16 http request count:2 blk:0
2023/07/31 15:14:22 [debug] 522#0: *16 http2 frame complete pos:0000562D049D191B end:0000562D049D191B
2023/07/31 15:14:22 [debug] 522#0: *16 http2 read handler
2023/07/31 15:14:22 [debug] 522#0: *16 SSL_read: 48
2023/07/31 15:14:22 [debug] 522#0: *16 SSL_read: -1
2023/07/31 15:14:22 [debug] 522#0: *16 SSL_get_error: 2
2023/07/31 15:14:22 [debug] 522#0: *16 http2 frame type:0 f:1 l:39 sid:1
2023/07/31 15:14:22 [debug] 522#0: *16 http2 DATA frame
2023/07/31 15:14:22 [debug] 522#0: *16 malloc: 0000562D04A15910:65536
2023/07/31 15:14:22 [debug] 522#0: *16 http2 frame complete pos:0000562D049D1920 end:0000562D049D1920

[chobits]

for http2 & upstream:

• read upstream and send data to downstream: ngx_http_upstream_process_non_buffered_request
• read downstream and send data to upstream: ngx_http_upstream_read_request_handler->ngx_http_upstream_send_request

[chobits]

Unfortunately, there is no low-level IO operation API of c->recv()/send() provided in nginx's HTTPv2 implementation. So if we want to support http/2 CONNECT tunnel, we need to use request_body API for reading DATA frames from http/2 request and output_filter for sending DATA frames through the http/2 CONNECT tunnel.

This makes the implementation of supporting HTTP/2 in this module more complex.

[doublex]

Great feature!! Thanks for your efforts!

[doublex]

Question: Is the hard part downstream or upstream? Would it be easier if only the downstream-connection is http2 (so eavesdroppers think it is http2)? E.g.: Client -> http2:connect -> nginx:chobits -> http11 -> website

[chobits]

Question: Is the hard part downstream or upstream? Would it be easier if only the downstream-connection is http2 (so eavesdroppers think it is http2)? E.g.: Client -> http2:connect -> nginx:chobits -> http11 -> website

This pr is under development, so some part of the logic is not completed. so it could not support the HTTP2 now.