I think this is one of those edge cases where it could be interpreted as either a bug or a missing feature. What's happening is this:
When calling Requests::request_multiple() with multiple requests, curl_multi_exec() will process them asynchronously in the background. Requests then loops over the CurlMultiHandle to handle the requests that finished, potentially out of order. So far so good.
The effect of this is that currently redirects are processed synchronously when encountered after every asynchronous request, which in a worst case scenario, where every request passed to Requests::request_multiple() leads to a redirect and $options['follow_redirects'] is true, essentially causes all requests to be processed near synchronously.
Describe the solution you'd like
I'd think the expectation of calling Requests::request_multiple() is that the requests are handled asynchronously, including potential redirects. What I don't know is the best way to achieve that.
I suppose the most ideal would be if any redirect adds a handle to the active CurlMultiHandle so we can continue processing them in the existing loop. I'm not sure that's possible, however (quite likely it's not). So the next best thing would be a new, separate call to curl_multi_exec(), which at least causes the redirect to be put in the background so we could choose to process the other responses first before coming back to this one.
The challenge here is that this too should then trigger all the usual hooks (requests.before_redirect_check, requests.before_redirect).
Describe alternatives you've considered
Another option would be to handle redirects in the Requests class by collecting all responses that indicate a redirect and issuing a new call to Curl::request_multiple() for those. This would be easier to do I suppose, at the cost of less concurrency.
[x] I intend to create a pull request to implement this feature myself.
Is your feature request related to a problem?
I think this is one of those edge cases where it could be interpreted as either a bug or a missing feature. What's happening is this:
When calling
Requests::request_multiple()with multiple requests,curl_multi_exec()will process them asynchronously in the background. Requests then loops over theCurlMultiHandleto handle the requests that finished, potentially out of order. So far so good.In the
Curltransport, when processing the finished request, it calls thetransport.internal.parse_responsehook, which theRequestsclass hooks into in https://github.com/WordPress/Requests/blob/develop/src/Requests.php#L531 or https://github.com/WordPress/Requests/blob/develop/src/Requests.php#L531. This leads toRequests::parse_response()call, which also handles redirects by callingRequests::request().The effect of this is that currently redirects are processed synchronously when encountered after every asynchronous request, which in a worst case scenario, where every request passed to
Requests::request_multiple()leads to a redirect and$options['follow_redirects']istrue, essentially causes all requests to be processed near synchronously.Describe the solution you'd like
I'd think the expectation of calling
Requests::request_multiple()is that the requests are handled asynchronously, including potential redirects. What I don't know is the best way to achieve that. I suppose the most ideal would be if any redirect adds a handle to the activeCurlMultiHandleso we can continue processing them in the existing loop. I'm not sure that's possible, however (quite likely it's not). So the next best thing would be a new, separate call tocurl_multi_exec(), which at least causes the redirect to be put in the background so we could choose to process the other responses first before coming back to this one. The challenge here is that this too should then trigger all the usual hooks (requests.before_redirect_check,requests.before_redirect).Describe alternatives you've considered
Another option would be to handle redirects in the
Requestsclass by collecting all responses that indicate a redirect and issuing a new call toCurl::request_multiple()for those. This would be easier to do I suppose, at the cost of less concurrency.I can try at least :slightly_smiling_face: