http.rb normalises URLs before making a request. This normalisation may reformat a perfectly valid URL. Even though the normalised URL in theory is equivalent to the original URL, some servers may not recognise this (see e.g. https://github.com/mastodon/mastodon/pull/26219).
Some of these problems are due to a bug in Addressable (see https://github.com/sporkmonger/addressable/issues/366). But even with that bug fixes, there is no reason to normalise the URL before fetching it, apart from fixing obviously invalid aspects such as spaces in the path. There is no benefit, only potential problems.
I suggest we take a more conservative approach and only escape characters in path and query outside the printable ASCII range, and never decode any existing percent-escape sequences. This is the approach taken by cURL, so that seems like a safe bet.
If somebody wants more “aggressive” normalisation for better caching in HTTP proxies or whatever, and they are aware of the risks connected with this, they can provide a custom normaliser or just call Addressable::URI#normalize before invoking htp.rb. But I don't think such potentially risky behaviour should be the default.
c960657
commented
1 year ago
http.rb normalises URLs before making a request. This normalisation may reformat a perfectly valid URL. Even though the normalised URL in theory is equivalent to the original URL, some servers may not recognise this (see e.g. https://github.com/mastodon/mastodon/pull/26219).
Some of these problems are due to a bug in Addressable (see https://github.com/sporkmonger/addressable/issues/366). But even with that bug fixes, there is no reason to normalise the URL before fetching it, apart from fixing obviously invalid aspects such as spaces in the path. There is no benefit, only potential problems.
I suggest we take a more conservative approach and only escape characters in path and query outside the printable ASCII range, and never decode any existing percent-escape sequences. This is the approach taken by cURL, so that seems like a safe bet.
If somebody wants more “aggressive” normalisation for better caching in HTTP proxies or whatever, and they are aware of the risks connected with this, they can provide a custom normaliser or just call
Addressable::URI#normalizebefore invoking htp.rb. But I don't think such potentially risky behaviour should be the default.Fixes #654