MHD adds duplicate Content-Length header in response when the header exists

kingo55 commented 2 years ago

Expected behaviour

If the response headers contain a Content-Length header then MHD doesn't add one.

Actual behaviour

When a response already contains a Content-Length header, MHD adds another one, causing some useragents to reject the response.

For example, when a non-compliant duplicate header exists, Python's aiohttp library, for example, raises an exception:

Traceback (most recent call last):
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/aiohttp/client_proto.py", line 213, in data_received
    messages, upgraded, tail = self._parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 551, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message='Duplicate Content-Length'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/jsonrpc_async/jsonrpc.py", line 33, in send_message
    response = await self._request(data=message.serialize())
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/aiohttp/client.py", line 559, in _request
    await resp.start(conn)
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 900, in start
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Duplicate Content-Length', url=URL('http://192.168.0.100:8080/jsonrpc')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/pykodi/kodi.py", line 164, in ping
    response = await self._server.JSONRPC.Ping()
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/jsonrpc_async/jsonrpc.py", line 35, in send_message
    raise TransportError('Transport Error', message, exc)
jsonrpc_base.jsonrpc.TransportError: ("Error calling method 'JSONRPC.Ping': Transport Error", ClientResponseError(RequestInfo(url=URL('http://192.168.0.100:8080/jsonrpc'), method='POST', headers=<CIMultiDictProxy('Host': '192.168.0.100:8080', 'Content-Type': 'application/json', 'Accept': 'application/json-rpc', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1', 'Authorization': 'Basic ******', 'Content-Length': '104')>, real_url=URL('http://192.168.0.100:8080/jsonrpc')), (), status=400, message='Duplicate Content-Length'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/robertkingston/Documents/pykodi-testing/test.py", line 19, in <module>
    asyncio.run(test())
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/robertkingston/Documents/pykodi-testing/test.py", line 11, in test
    await kodi.ping()
  File "/Users/robertkingston/Documents/pykodi-testing/venv/lib/python3.9/site-packages/pykodi/kodi.py", line 170, in ping
    raise CannotConnectError from error
pykodi.kodi.CannotConnectError

Further details

I discovered this behaviour downstream in XBMC which uses a MHD version 0x00097500 - details here: https://github.com/xbmc/xbmc/issues/20759

We think it may be due to this commit https://github.com/Karlson2k/libmicrohttpd/commit/4820d93bc91eea924dcad29e879f31d8d28649a1.

As the resolution to this issue is external to Kodi, I thought I should raise this in the correct upstream repo.

Karlson2k commented 2 years ago

Kodi incorrectly uses MHD API. MHD automatically sets "Content-Length" header when needed (and allowed by RFC). Flag MHD_RF_INSANITY_HEADER_CONTENT_LENGTH breaks standard processing and should not be used unless application wants to produce non-standard response not compatible with RFC specifications.

PR with fix for Kodi: https://github.com/xbmc/xbmc/pull/21019

kingo55 commented 2 years ago

Thanks for the API fix on Kodi, @Karlson2k! Really appreciate it - MHD is the glue between Home Assistant and Kodi.

Just saved potentially a few thousand Home Assistant / Kodi users' smart homes from breaking.

Karlson2k commented 2 years ago

@kingo55 Feel free to contract me directly by mail or in MHD mailing list for any MHD-related issues.

Karlson2k / libmicrohttpd