mplubratt
commented
1 year ago Buried in the hackney documentation, I found a single comment that follow_redirects are only followed when the response is received synchronously.
line 103-104 https://github.com/benoitc/hackney/blob/3e2b578d5d42534cc57f43749af6e8016f8e80fb/src/hackney.erl
Since this seems to be a hackney bug/"feature", my work around is to synchronously check the header for a 307 status first. If it is, follow the Location field in the header.
Documentation should probably be updated to reflect the hackney comment.
I'm writing a streaming processor for some large files in a Hadoop cluster. The files are too big to keep in memory, so I want to use async get! requests into a Stream.resource. The Hadoop cluster will provide a redirect from the name node to the cluster node with the file actually on it. However, follow_redirect doesn't seem to work with an async request.
If I do an async request without follow_redirect, I get back a %HTTPoison.AsyncStatus, %HTTPoison.AsyncHeaders and %HTTPoison.AsyncEnd messages showing the 307 status along with the redirect "Location" in the headers.
If I do an async request with follow_redirect, I only get a %HTTPoison.AsyncRedirect. There is no %HTTPoison.AsyncStatus message with a 307, or a %HTTPoison.AsyncEnd, only the redirect. There are no other messages in the mailbox, and the request doesn't follow the redirect even though it's a GET, and the status is 307.