Failed to open stream: Too many open files in Requests.php

[EBGTM]

While using Requests in a script that shoots out multiple thousands of requests, I get the following error:

PHP Warning:  require_once(/.../Requests/Exception.php): failed to open stream: Too many open files in /.../Requests.php on line 136
PHP Fatal error:  require_once(): Failed opening required '/.../Requests/Exception.php' (include_path='.:/usr/local/lib/php:/var/www/...') in /.../Requests.php on line 136

I believe this indicates that file handles are not being closed properly.

NOTE: This happens when using the cURL transport only, switching to fsockopen mitigated the issue.

[soulseekah]

Are you able to recollect what the actual request parameters were? Method, options, sample of posted data, etc.?

[EBGTM]

It has been a while since I reported this issue, but I was able to recall the context. The best I can give you is the class method used in the script I mentioned. Here is the full method (note that there is also a GET variant that is very similar):

/**
 * A function that shoots out a POST request to X API
 *
 * @param $endpoint
 * @param $params
 * @param bool | array $debug - false when turned off, array of options when on
 * @return array
 */
private function _post($endpoint, $params, $debug = false) {
    $return = array('success' => false);
    $response = null;
    $attempts = 5;
    do {
        try {
            $response = Requests::post($this->_url($endpoint), ['Content-Type' => 'application/json'], json_encode($params), ['transport' => 'Requests_Transport_fsockopen']);
        } catch(Exception $e) {
            $attempts--;
            sleep(1);
        }
    } while(empty($response) && $attempts > 0);

    if(!empty($response) && $response->success) {
        if($debug) {$this->_dump($response->body, $debug['dumpAsJson']); exit;}
        $body = json_decode($response->body, true);
        if($body['success']) {
            $return['success'] = true;
            $return['data'] = $body['data'];
        } else {
            $return['errorMessage'] = $body['errorMessage'];
        }
    }
    if($debug) {$this->_dump($response); exit;}
    return $return;
}

The context here is that we are using this class to query an external API through a regularly run cron script. The script, at the time, would query this API thousands of times over the course of 6-12 hours (probably around 5000 requests total). The long run time wasn't due to the requests themselves but the processing that occurred after each request, so it wasn't a timeout related issue.

The script has run smoothly ever since the transport parameter was set to Requests_Transport_fsockopen as I mentioned in my original post.

[soulseekah]

Interesting. Thanks.

PHP automatically runs the resource destructor as soon as all references to that resource are dropped.

In the current master version I've been able to reproduce some issues with unclosed file descriptor limits and out of memory limits in the cURL transport by using hooks to throw exceptions in irregular places. Namely in the curl.after_send and curl.after_request hooks and only if the filename option was used (a reference to it is retained so not freed on exception), I could also do it by hooking into curl.before_request and keeping a reference to the cURL handle around.

Your example, however, does not show signs of you using either hooks or the write-to-file functionality. So I'm stumped.

Do you remember which PHP version you were on back then and which version of Requests?

[EBGTM]

Yeah, I'm stumped along with you. At the time, I remember trying all sorts of things and the only thing that worked was changing the transport altogether.

Anyway, I don't believe we've upgraded PHP on that server since and requests has not been updated either.

PHP version: 5.6.27 Requests version: 1.6