HTTPRequests do not correctly handle URL fragments

[bruce-hill]

Tested versions

• Reproducible in v4.3.dev6.official [89850d553], v4.2.1.stable.official [b09f793f5]

System information

Windows 10

Issue description

If you make an HTTP request in Godot using a URL that contains a URI Fragment, Godot does not give any error messages and creates an HTTP request that includes the URI fragment, which is not supposed to be sent to the server. Many servers will not accept requests sent to paths that include a fragment and will return a 400 or 404 response. The correct behavior should be to strip the fragment out of the URL, which is what tools like cURL do.

Steps to reproduce

1. Call HTTPRequest.request(url) with a URL that has #whatever at the end.
2. The HTTP request that gets sent includes the fragment when it should not.

To verify this locally, I ran netcat (nc -lk 12345) and made a request to $HTTPRequest.request("http://127.0.0.1:12345/page#fragment"). The netcat output was:

GET /page#fragment HTTP/1.1
Host: 127.0.0.1:12345
User-Agent: GodotEngine/4.3.dev6.official (Windows)
Accept: */*

For comparison, HTTP requests sent by curl do not include the fragment: curl 'http://127.0.0.1:12345/page#fragment

GET /page HTTP/1.1
Host: 127.0.0.1:12345
User-Agent: curl/7.68.0
Accept: */*

Minimal reproduction project (MRP)

HTTPFragmentIssue.zip

[Calinou]

As a workaround, url.split("#")[0] should get the job done to strip the fragment part of the URL.

[MikkBenelis]

I faced the same issue about two weeks ago, being struggling with it until found the root cause today and surprised that it'll be solved. Just wanted to say that it can lead to potential issues in future. I think, most web servers (at least Google's one) can handle this fragment part and process or just ignore it. Imagine you had a link www.abc.com/qwe#asd, then it was moved to www.abc.com/qwert#asdfg and you want to apply a 301 redirect to it. If the HTTPRequest will ignore the fragment part, it'll not be able to follow the redirect link properly. Despite this issue, it is better to ignore it since some of the web servers will just respond with the 400 status code when find the number sign in URI.