Streaming URLs hang HTTPoison

[mfclarke]

Hi,

I'm using HTTPoison to fetch html data from arbitrary URLs. But if a URL points to a stream, HTTPoison never completes the standard HTTPoison.get, and there's no completion timeout I can use (that I know of), since timeout is just for the initial connection and recv_timeout is for the beginning of the response (not the completion of the response).

Is there a way I can detect if a URL points to a stream with no size, rather than a response with a fixed size, so I can skip them? Or failing that, is there an option I can pass that will timeout of the request isn't completed in x seconds?

Thanks!

[edgurgel]

Could you point me to an example on how to replicate this issue? I suppose we can expose a "cancel_request" as hackney has ways to do this.

[mfclarke]

Sure, you can execute the following:

HTTPoison.get("http://193.34.51.165/917xfm_128", [], [timeout: 2500, recv_timeout: 7000])

Replace the url with any http streaming url, and the result will be the same.

[mfclarke]

Just a follow up on this old issue @edgurgel, I had a chance to attempt this with stream_to so I could try doing :hackney.close(ref) to abort the fetch if I accidentally hit an endlessly streaming URL.

But unfortunately this leaks HTTPoison processes. They get stuck in transformer/5.

Not sure where to go from here, but any help would be much appreciated!

[Fitblip]

Hey sorry for resurrecting a really old thread, but I ran into this recently with HTTPoison and streaming large files (to download large files).

I found that disabling the hackney pooling (hackney: [pool: false]) seems to resolve my issues:

HTTPoison.get!(
    url, 
    ["Authorization": "Bearer #{token}"], 
    [stream_to: self(), async: :once, hackney: [pool: false]]
)

I'm going to update the other thread with the same (#174 hopefully that's OK).