The HTTP parser in AIOHTTP has numerous problems with header parsing, which could lead to request smuggling.
This parser is only used when AIOHTTP_NO_EXTENSIONS is enabled (or not using a prebuilt wheel).
Details
Bug 1: Bad parsing of Content-Length values
Description
RFC 9110 says this:
Content-Length = 1*DIGIT
AIOHTTP does not enforce this rule, presumably because of an incorrect usage of the builtin int constructor. Because the int constructor accepts + and - prefixes, and digit-separating underscores, using int to parse CL values leads AIOHTTP to significant misinterpretation.
Examples
GET / HTTP/1.1\r\n
Content-Length: -0\r\n
\r\n
X
GET / HTTP/1.1\r\n
Content-Length: +0_1\r\n
\r\n
X
Suggested action
Verify that a Content-Length value consists only of ASCII digits before parsing, as the standard requires.
Bug 2: Improper handling of NUL, CR, and LF in header values
Description
RFC 9110 says this:
Field values containing CR, LF, or NUL characters are invalid and dangerous, due to the varying ways that implementations might parse and interpret those characters; a recipient of CR, LF, or NUL within a field value MUST either reject the message or replace each of those characters with SP before further processing or forwarding of that message.
AIOHTTP's HTTP parser does not enforce this rule, and will happily process header values containing these three forbidden characters without replacing them with SP.
Examples
GET / HTTP/1.1\r\n
Header: v\x00alue\r\n
\r\n
GET / HTTP/1.1\r\n
Header: v\ralue\r\n
\r\n
GET / HTTP/1.1\r\n
Header: v\nalue\r\n
\r\n
Suggested action
Reject all messages with NUL, CR, or LF in a header value. The translation to space thing, while technically allowed, does not seem like a good idea to me.
Bug 3: Improper stripping of whitespace before colon in HTTP headers
Description
RFC 9112 says this:
No whitespace is allowed between the field name and colon. In the past, differences in the handling of such whitespace have led to security vulnerabilities in request routing and response handling. A server MUST reject, with a response status code of 400 (Bad Request), any received request message that contains whitespace between a header field name and colon.
AIOHTTP does not enforce this rule, and will simply strip any whitespace before the colon in an HTTP header.
Example
GET / HTTP/1.1\r\n
Content-Length : 1\r\n
\r\n
X
Suggested action
Reject all messages with whitespace before a colon in a header field, as the standard requires.
PoC
Example requests are embedded in the previous section. To reproduce these bugs, start an AIOHTTP server without llhttp (i.e. AIOHTTP_NO_EXTENSIONS=1) and send the requests given in the previous section. (e.g. by printfing into nc)
Impact
Each of these bugs can be used for request smuggling.
llhttp 8.1.1 is vulnerable to two request smuggling vulnerabilities.
Details have not been disclosed yet, so refer to llhttp for future information.
The issue is resolved by using llhttp 9+ (which is included in aiohttp 3.8.6+).
Improper validation makes it possible for an attacker to modify the HTTP request (e.g. insert a new header) or even create a new HTTP request if the attacker controls the HTTP method.
Details
The vulnerability occurs only if the attacker can control the HTTP method (GET, POST etc.) of the request.
Previous releases performed no validation on the provided value. If an attacker controls the HTTP method it will be used as is and can lead to HTTP request smuggling.
If the attacker can control the HTTP version of the request it will be able to modify the request (request smuggling).
Workaround
If unable to upgrade and using user-provided values for the request method, perform manual validation of the user value (e.g. by restricting it to a few known values like GET, POST etc.).
Improper validation make it possible for an attacker to modify the HTTP request (e.g. to insert a new header) or even create a new HTTP request if the attacker controls the HTTP version.
Details
The vulnerability only occurs if the attacker can control the HTTP version of the request (including its type).
For example if an unvalidated JSON value is used as a version and the attacker is then able to pass an array as the version parameter.
Furthermore, the vulnerability only occurs when the Connection header is passed to the headers parameter.
At this point, the library will use the parsed value to create the request. If a list is passed, then it bypasses validation and it is possible to perform CRLF injection.
If these specific conditions are met and you are unable to upgrade, then validate the user input to the version parameter to ensure it is a str.
Release Notes
aio-libs/aiohttp (aiohttp)
### [`v3.9.0`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#390-2023-11-18)
[Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.6...v3.9.0)
\==================
## Features
- Introduced `AppKey` for static typing support of `Application` storage.
See https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config
`#5864 `\_
- Added a graceful shutdown period which allows pending tasks to complete before the application's cleanup is called.
The period can be adjusted with the `shutdown_timeout` parameter. -- by :user:`Dreamsorcerer`.
See https://docs.aiohttp.org/en/latest/web_advanced.html#graceful-shutdown
`#7188 `\_
- Added `handler_cancellation `\_ parameter to cancel web handler on client disconnection. -- by :user:`mosquito`
This (optionally) reintroduces a feature removed in a previous release.
Recommended for those looking for an extra level of protection against denial-of-service attacks.
`#7056 `\_
- Added support for setting response header parameters `max_line_size` and `max_field_size`.
`#2304 `\_
- Added `auto_decompress` parameter to `ClientSession.request` to override `ClientSession._auto_decompress`. -- by :user:`Daste745`
`#3751 `\_
- Changed `raise_for_status` to allow a coroutine.
`#3892 `\_
- Added client brotli compression support (optional with runtime check).
`#5219 `\_
- Added `client_max_size` to `BaseRequest.clone()` to allow overriding the request body size. -- :user:`anesabml`.
`#5704 `\_
- Added a middleware type alias `aiohttp.typedefs.Middleware`.
`#5898 `\_
- Exported `HTTPMove` which can be used to catch any redirection request
that has a location -- :user:`dreamsorcerer`.
`#6594 `\_
- Changed the `path` parameter in `web.run_app()` to accept a `pathlib.Path` object.
`#6839 `\_
- Performance: Skipped filtering `CookieJar` when the jar is empty or all cookies have expired.
`#7819 `\_
- Performance: Only check origin if insecure scheme and there are origins to treat as secure, in `CookieJar.filter_cookies()`.
`#7821 `\_
- Performance: Used timestamp instead of `datetime` to achieve faster cookie expiration in `CookieJar`.
`#7824 `\_
- Added support for passing a custom server name parameter to HTTPS connection.
`#7114 `\_
- Added support for using Basic Auth credentials from :file:`.netrc` file when making HTTP requests with the
:py:class:`~aiohttp.ClientSession` `trust_env` argument is set to `True`. -- by :user:`yuvipanda`.
`#7131 `\_
- Turned access log into no-op when the logger is disabled.
`#7240 `\_
- Added typing information to `RawResponseMessage`. -- by :user:`Gobot1234`
`#7365 `\_
- Removed `async-timeout` for Python 3.11+ (replaced with `asyncio.timeout()` on newer releases).
`#7502 `\_
- Added support for `brotlicffi` as an alternative to `brotli` (fixing Brotli support on PyPy).
`#7611 `\_
- Added `WebSocketResponse.get_extra_info()` to access a protocol transport's extra info.
`#7078 `\_
- Allow `link` argument to be set to None/empty in HTTP 451 exception.
`#7689 `\_
## Bugfixes
- Implemented stripping the trailing dots from fully-qualified domain names in `Host` headers and TLS context when acting as an HTTP client.
This allows the client to connect to URLs with FQDN host name like `https://example.com./`.
\-- by :user:`martin-sucha`.
`#3636 `\_
- Fixed client timeout not working when incoming data is always available without waiting. -- by :user:`Dreamsorcerer`.
`#5854 `\_
- Fixed `readuntil` to work with a delimiter of more than one character.
`#6701 `\_
- Added `__repr__` to `EmptyStreamReader` to avoid `AttributeError`.
`#6916 `\_
- Fixed bug when using `TCPConnector` with `ttl_dns_cache=0`.
`#7014 `\_
- Fixed response returned from expect handler being thrown away. -- by :user:`Dreamsorcerer`
`#7025 `\_
- Avoided raising `UnicodeDecodeError` in multipart and in HTTP headers parsing.
`#7044 `\_
- Changed `sock_read` timeout to start after writing has finished, avoiding read timeouts caused by an unfinished write. -- by :user:`dtrifiro`
`#7149 `\_
- Fixed missing query in tracing method URLs when using `yarl` 1.9+.
`#7259 `\_
- Changed max 32-bit timestamp to an aware datetime object, for consistency with the non-32-bit one, and to avoid a `DeprecationWarning` on Python 3.12.
`#7302 `\_
- Fixed `EmptyStreamReader.iter_chunks()` never ending. -- by :user:`mind1m`
`#7616 `\_
- Fixed a rare `RuntimeError: await wasn't used with future` exception. -- by :user:`stalkerg`
`#7785 `\_
- Fixed issue with insufficient HTTP method and version validation.
`#7700 `\_
- Added check to validate that absolute URIs have schemes.
`#7712 `\_
- Fixed unhandled exception when Python HTTP parser encounters unpaired Unicode surrogates.
`#7715 `\_
- Updated parser to disallow invalid characters in header field names and stop accepting LF as a request line separator.
`#7719 `\_
- Fixed Python HTTP parser not treating 204/304/1xx as an empty body.
`#7755 `\_
- Ensure empty body response for 1xx/204/304 per RFC 9112 sec 6.3.
`#7756 `\_
- Fixed an issue when a client request is closed before completing a chunked payload. -- by :user:`Dreamsorcerer`
`#7764 `\_
- Edge Case Handling for ResponseParser for missing reason value.
`#7776 `\_
- Fixed `ClientWebSocketResponse.close_code` being erroneously set to `None` when there are concurrent async tasks receiving data and closing the connection.
`#7306 `\_
- Added HTTP method validation.
`#6533 `\_
- Fixed arbitrary sequence types being allowed to inject values via version parameter. -- by :user:`Dreamsorcerer`
`#7835 `\_
- Performance: Fixed increase in latency with small messages from websocket compression changes.
`#7797 `\_
## Improved Documentation
- Fixed the `ClientResponse.release`'s type in the doc. Changed from `comethod` to `method`.
`#5836 `\_
- Added information on behavior of base_url parameter in `ClientSession`.
`#6647 `\_
- Fixed `ClientResponseError` docs.
`#6700 `\_
- Updated Redis code examples to follow the latest API.
`#6907 `\_
- Added a note about possibly needing to update headers when using `on_response_prepare`. -- by :user:`Dreamsorcerer`
`#7283 `\_
- Completed `trust_env` parameter description to honor `wss_proxy`, `ws_proxy` or `no_proxy` env.
`#7325 `\_
- Expanded SSL documentation with more examples (e.g. how to use certifi). -- by :user:`Dreamsorcerer`
`#7334 `\_
- Fix, update, and improve client exceptions documentation.
`#7733 `\_
## Deprecations and Removals
- Added `shutdown_timeout` parameter to `BaseRunner`, while
deprecating `shutdown_timeout` parameter from `BaseSite`. -- by :user:`Dreamsorcerer`
`#7718 `\_
- Dropped Python 3.6 support.
`#6378 `\_
- Dropped Python 3.7 support. -- by :user:`Dreamsorcerer`
`#7336 `\_
- Removed support for abandoned `tokio` event loop. -- by :user:`Dreamsorcerer`
`#7281 `\_
## Misc
- Made `print` argument in `run_app()` optional.
`#3690 `\_
- Improved performance of `ceil_timeout` in some cases.
`#6316 `\_
- Changed importing Gunicorn to happen on-demand, decreasing import time by ~53%. -- :user:`Dreamsorcerer`
`#6591 `\_
- Improved import time by replacing `http.server` with `http.HTTPStatus`.
`#6903 `\_
- Fixed annotation of `ssl` parameter to disallow `True`. -- by :user:`Dreamsorcerer`.
`#7335 `\_
***
### [`v3.8.6`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#386-2023-10-07)
[Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.5...v3.8.6)
\==================
## Security bugfixes
- Upgraded the vendored copy of llhttp\_ to v9.1.3 -- by :user:`Dreamsorcerer`
Thanks to :user:`kenballus` for reporting this, see
https://github.com/aio-libs/aiohttp/security/advisories/GHSA-pjjw-qhg8-p2p9.
.. \_llhttp: https://llhttp.org
`#7647 `\_
- Updated Python parser to comply with RFCs 9110/9112 -- by :user:`Dreamorcerer`
Thanks to :user:`kenballus` for reporting this, see
https://github.com/aio-libs/aiohttp/security/advisories/GHSA-gfw2-4jvh-wgfg.
`#7663 `\_
## Deprecation
- Added `fallback_charset_resolver` parameter in `ClientSession` to allow a user-supplied
character set detection function.
Character set detection will no longer be included in 3.9 as a default. If this feature is needed,
please use `fallback_charset_resolver `\_.
`#7561 `\_
## Features
- Enabled lenient response parsing for more flexible parsing in the client
(this should resolve some regressions when dealing with badly formatted HTTP responses). -- by :user:`Dreamsorcerer`
`#7490 `\_
## Bugfixes
- Fixed `PermissionError` when `.netrc` is unreadable due to permissions.
`#7237 `\_
- Fixed output of parsing errors pointing to a `\n`. -- by :user:`Dreamsorcerer`
`#7468 `\_
- Fixed `GunicornWebWorker` max_requests_jitter not working.
`#7518 `\_
- Fixed sorting in `filter_cookies` to use cookie with longest path. -- by :user:`marq24`.
`#7577 `\_
- Fixed display of `BadStatusLine` messages from llhttp\_. -- by :user:`Dreamsorcerer`
`#7651 `\_
***
Configuration
📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
â™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
[ ] If you want to rebase/retry this PR, check this box
This PR contains the following updates:
==3.8.5->==3.9.0GitHub Vulnerability Alerts
CVE-2023-47627
Summary
The HTTP parser in AIOHTTP has numerous problems with header parsing, which could lead to request smuggling. This parser is only used when
AIOHTTP_NO_EXTENSIONSis enabled (or not using a prebuilt wheel).Details
Bug 1: Bad parsing of
Content-LengthvaluesDescription
RFC 9110 says this:
AIOHTTP does not enforce this rule, presumably because of an incorrect usage of the builtin
intconstructor. Because theintconstructor accepts+and-prefixes, and digit-separating underscores, usingintto parse CL values leads AIOHTTP to significant misinterpretation.Examples
Suggested action
Verify that a
Content-Lengthvalue consists only of ASCII digits before parsing, as the standard requires.Bug 2: Improper handling of NUL, CR, and LF in header values
Description
RFC 9110 says this:
AIOHTTP's HTTP parser does not enforce this rule, and will happily process header values containing these three forbidden characters without replacing them with SP.
Examples
Suggested action
Reject all messages with NUL, CR, or LF in a header value. The translation to space thing, while technically allowed, does not seem like a good idea to me.
Bug 3: Improper stripping of whitespace before colon in HTTP headers
Description
RFC 9112 says this:
AIOHTTP does not enforce this rule, and will simply strip any whitespace before the colon in an HTTP header.
Example
Suggested action
Reject all messages with whitespace before a colon in a header field, as the standard requires.
PoC
Example requests are embedded in the previous section. To reproduce these bugs, start an AIOHTTP server without llhttp (i.e.
AIOHTTP_NO_EXTENSIONS=1) and send the requests given in the previous section. (e.g. byprintfing intonc)Impact
Each of these bugs can be used for request smuggling.
GHSA-pjjw-qhg8-p2p9
Summary
llhttp 8.1.1 is vulnerable to two request smuggling vulnerabilities. Details have not been disclosed yet, so refer to llhttp for future information. The issue is resolved by using llhttp 9+ (which is included in aiohttp 3.8.6+).
CVE-2023-49082
Summary
Improper validation makes it possible for an attacker to modify the HTTP request (e.g. insert a new header) or even create a new HTTP request if the attacker controls the HTTP method.
Details
The vulnerability occurs only if the attacker can control the HTTP method (GET, POST etc.) of the request.
Previous releases performed no validation on the provided value. If an attacker controls the HTTP method it will be used as is and can lead to HTTP request smuggling.
PoC
A minimal example can be found here: https://gist.github.com/jnovikov/7f411ae9fe6a9a7804cf162a3bdbb44b
Impact
If the attacker can control the HTTP version of the request it will be able to modify the request (request smuggling).
Workaround
If unable to upgrade and using user-provided values for the request method, perform manual validation of the user value (e.g. by restricting it to a few known values like GET, POST etc.).
CVE-2023-49081
Summary
Improper validation make it possible for an attacker to modify the HTTP request (e.g. to insert a new header) or even create a new HTTP request if the attacker controls the HTTP version.
Details
The vulnerability only occurs if the attacker can control the HTTP version of the request (including its type). For example if an unvalidated JSON value is used as a version and the attacker is then able to pass an array as the
versionparameter. Furthermore, the vulnerability only occurs when theConnectionheader is passed to theheadersparameter.At this point, the library will use the parsed value to create the request. If a list is passed, then it bypasses validation and it is possible to perform CRLF injection.
PoC
The POC below shows an example of providing an unvalidated array as a version: https://gist.github.com/jnovikov/184afb593d9c2114d77f508e0ccd508e
Impact
CRLF injection leading to Request Smuggling.
Workaround
If these specific conditions are met and you are unable to upgrade, then validate the user input to the
versionparameter to ensure it is astr.Release Notes
aio-libs/aiohttp (aiohttp)
### [`v3.9.0`](https://togithub.com/aio-libs/aiohttp/blob/HEAD/CHANGES.rst#390-2023-11-18) [Compare Source](https://togithub.com/aio-libs/aiohttp/compare/v3.8.6...v3.9.0) \================== ## Features - Introduced `AppKey` for static typing support of `Application` storage. See https://docs.aiohttp.org/en/stable/web_advanced.html#application-s-config `#5864Configuration
📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
â™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Renovate Bot.