Any HTTP/2 request fails on current HTTPie versions - Connection-specific header field present: b'connection'

[00dani]

This issue occurs both on the current stable HTTPie release, 0.9.9, and on the bleeding-edge development version de38f86 installed using pip3 install --upgrade https://github.com/jkbrzt/httpie/archive/master.tar.gz.

After installing httpie-http2 with pip3 install httpie-http2, every attempt to make an HTTP/2 request will fail, printing the error message http: error: ProtocolError: Connection-specific header field present: b'connection'.

The full --debug output is:

$ http https://google.com --debug
HTTPie 0.9.9
Requests 2.13.0
Pygments 2.2.0
Python 3.6.0 (default, Dec 24 2016, 08:01:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
/usr/local/opt/python3/bin/python3.6
Darwin 16.5.0

<Environment {
    "colors": 256,
    "config": {
        "__meta__": {
            "about": "HTTPie configuration file",
            "help": "https://github.com/jkbrzt/httpie#config",
            "httpie": "1.0.0-dev"
        },
        "default_options": "[]"
    },
    "config_dir": "/Users/dani/.httpie",
    "is_windows": false,
    "stderr": "<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>",
    "stderr_isatty": true,
    "stdin": "<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>",
    "stdin_encoding": "UTF-8",
    "stdin_isatty": true,
    "stdout": "<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>",
    "stdout_encoding": "UTF-8",
    "stdout_isatty": true
}>

>>> requests.request(**{
    "allow_redirects": false,
    "auth": "None",
    "cert": "None",
    "data": {},
    "files": {},
    "headers": {
        "User-Agent": "HTTPie/0.9.9"
    },
    "method": "get",
    "params": {},
    "proxies": {},
    "stream": true,
    "timeout": 30,
    "url": "https://google.com",
    "verify": true
})

http: error: ProtocolError: Connection-specific header field present: b'connection'.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/hyper/common/connection.py", line 103, in request
    method=method, url=url, body=body, headers=headers
  File "/usr/local/lib/python3.6/site-packages/hyper/http11/connection.py", line 169, in request
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/hyper/http11/connection.py", line 130, in connect
    raise TLSUpgrade(proto, sock)
hyper.common.exceptions.TLSUpgrade

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/http", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/httpie/__main__.py", line 11, in main
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/httpie/core.py", line 227, in main
    log_error=log_error,
  File "/usr/local/lib/python3.6/site-packages/httpie/core.py", line 99, in program
    final_response = get_response(args, config_dir=env.config.directory)
  File "/usr/local/lib/python3.6/site-packages/httpie/client.py", line 70, in get_response
    response = requests_session.request(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/hyper/contrib.py", line 78, in send
    request.headers
  File "/usr/local/lib/python3.6/site-packages/hyper/common/connection.py", line 121, in request
    method=method, url=url, body=body, headers=headers
  File "/usr/local/lib/python3.6/site-packages/hyper/http20/connection.py", line 281, in request
    self.endheaders(message_body=body, final=True, stream_id=stream_id)
  File "/usr/local/lib/python3.6/site-packages/hyper/http20/connection.py", line 555, in endheaders
    stream.send_headers(headers_only)
  File "/usr/local/lib/python3.6/site-packages/hyper/http20/stream.py", line 98, in send_headers
    conn.send_headers(self.stream_id, headers, end_stream)
  File "/usr/local/lib/python3.6/site-packages/h2/connection.py", line 813, in send_headers
    headers, self.encoder, end_stream
  File "/usr/local/lib/python3.6/site-packages/h2/stream.py", line 788, in send_headers
    headers, encoder, hf, hdr_validation_flags
  File "/usr/local/lib/python3.6/site-packages/h2/stream.py", line 1119, in _build_headers_frames
    encoded_headers = encoder.encode(headers)
  File "/usr/local/lib/python3.6/site-packages/hpack/hpack.py", line 229, in encode
    for header in headers:
  File "/usr/local/lib/python3.6/site-packages/h2/utilities.py", line 368, in _validate_host_authority_header
    for header in headers:
  File "/usr/local/lib/python3.6/site-packages/h2/utilities.py", line 302, in _reject_pseudo_header_fields
    for header in headers:
  File "/usr/local/lib/python3.6/site-packages/h2/utilities.py", line 274, in _reject_connection_header
    "Connection-specific header field present: %s." % header[0]
h2.exceptions.ProtocolError: Connection-specific header field present: b'connection'.

HTTP/1.1 requests are unaffected even when httpie-http2 is installed, presumably because the plugin only does anything if you're actually trying to use HTTP/2, but all requests to HTTP/2 hosts become impossible.

[Lukasa]

@00dani Can you try running pip install -U git+https://github.com/Lukasa/hyper.git please? I want to see if the current master branch helps or hurts.

[00dani]

It helps! With HTTPie 0.9.9 and httpie-http2 0.0.1, installing the master branch of Hyper seems to fix this issue:

$ pip3 install -U git+https://github.com/Lukasa/hyper.git
Collecting git+https://github.com/Lukasa/hyper.git
  Cloning https://github.com/Lukasa/hyper.git to /private/var/folders/sj/13hkwxcd7s7fdc7hjdhltn9h0000gn/T/pip-npvgdq8f-build
Requirement already up-to-date: h2!=2.5.0,<3.0,>=2.4 in /usr/local/lib/python3.6/site-packages (from hyper==0.8.0.dev0)
Requirement already up-to-date: hyperframe<4.0,>=3.2 in /usr/local/lib/python3.6/site-packages (from hyper==0.8.0.dev0)
Requirement already up-to-date: hpack<3,>=2.2 in /usr/local/lib/python3.6/site-packages (from h2!=2.5.0,<3.0,>=2.4->hyper==0.8.0.dev0)
Installing collected packages: hyper
  Found existing installation: hyper 0.7.0
    Uninstalling hyper-0.7.0:
      Successfully uninstalled hyper-0.7.0
  Running setup.py install for hyper ... done
Successfully installed hyper-0.8.0.dev0
$ http https://google.com
HTTP/2 302

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.com.au/?gfe_rd=cr&amp;ei=xxSkWOTfD7PM8gfToaLYDA">here</A>.
</BODY></HTML>

The response headers are still hidden, but that's a problem for another ticket.

[Lukasa]

Yes, so I should note that the hyper library has not been a major focus for me recently, mostly because I'm working on a series of large and fairly important urllib3 changes. So there may be some issues around it.