Response body for redirects not ignored in streaming mode

[fredrikekre]

HTTP.jl do not ignore the response body in redirects as discovered here: https://discourse.julialang.org/t/unexpected-end-of-data-when-updating-registries-with-localpackageserver/112782. The is introduced in HTTP.jl 1.6 (1.0 - 1.5 is OK) so the bug is in this range: https://github.com/JuliaWeb/HTTP.jl/compare/v1.5.5...v1.6.3 which includes a number of patches related to redirects.

---

Sample server:

server = HTTP.listen!(8123) do http
    if http.message.target == "/200"
        HTTP.setstatus(http, 200)
        HTTP.startwrite(http)
        HTTP.write(http, "This is /200\n")
    else
        HTTP.setstatus(http, 301)
        HTTP.setheader(http, "Location" => "/200")
        HTTP.startwrite(http)
        HTTP.write(http, "Go to /200\n")
    end
    return
end

HTTP.jl:

julia> HTTP.get("http://localhost:8123"; redirect=false, response_stream = stderr);
Go to /200                                                           # <------

julia> HTTP.get("http://localhost:8123"; response_stream = stderr);
Go to /200                                                           # <------
This is /200                                                         # <------

curl:

$ curl -v http://localhost:8123
*   Trying 127.0.0.1:8123...
* Connected to localhost (127.0.0.1) port 8123 (#0)
> GET / HTTP/1.1
> Host: localhost:8123
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Location: /200
< Transfer-Encoding: chunked
<
Go to /200                                                           # <------
* Connection #0 to host localhost left intact

$ curl -Lv http://localhost:8123
*   Trying 127.0.0.1:8123...
* Connected to localhost (127.0.0.1) port 8123 (#0)
> GET / HTTP/1.1
> Host: localhost:8123
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Location: /200
< Transfer-Encoding: chunked
<
* Ignoring the response-body                                         # <------
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8123/200'
* Found bundle for host localhost: 0x55d6fea933c0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 8123 (#0)
> GET /200 HTTP/1.1
> Host: localhost:8123
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
<
This is /200                                                         # <------
* Connection #0 to host localhost left intact

[quinnj]

Ok, so you're saying the issue is that in the 2nd case, both the redirect body and the redirected body are both written to the response_stream? Yeah, I agree we don't want that behavior.

I've put up https://github.com/JuliaWeb/HTTP.jl/pull/1168 for now to see how CI looks; not sure if it may unintentionally break something else.

[fredrikekre]

Yea sorry if the description wasn't so clear. Basically, only the last body should be written.