Error: Unable to open <File-Path> using mode r: fopen(<File-Path>): failed to open stream: Too many open files

[prayagverma]

Thanks for creating this great plugin.

After setting up the plugin and testing it on small folders (using upload-directory command), it worked without problems.

But, when I tried to upload one of the folders (aka the 2020 folder under the uploads directory), I ran into the following error -

Error: Unable to open <File-Path> using mode r: fopen(<File-Path>): failed to open stream: Too many open files

I searched around and came across another issue (https://github.com/humanmade/S3-Uploads/issues/195) which appears of the same type but unfortunately, it doesn't contain many details

Not completely similar to this, but I also came across https://stackoverflow.com/questions/45663757/unable-to-open-using-mode-r-fopen-aws-elastic-beanstalk. As suggested in one of the answers, I did check POST_MAX_SIZE and UPLOAD_MAX_FILESIZE values in the php.ini file but those were set sufficiently high (1256M)

I wanted to add that when running the ulimit -n command, the output is 100. I am unable to change this value (as I am doing this on a shared hosting server)

[rmccue]

Thanks for the detailed issue!

It sounds like potentially the transfer is either opening too many files at once, or it's not correctly closing the files out when complete.

Does this occur if you set --concurrency=1 on the command line? Our default is 5; could be that's too high for your server (100 is fairly low).

We're using the AWS S3 Transfer Manager pretty directly here, and it doesn't seem like there's any other relevant config options for this.

My other suspicion is that maybe we're not correctly closing file handles internally. Could be possible.

[prayagverma]

Thanks for replying

When I set --concurrency=1, there is still an error but it is different from the previous one

The error I see is as follows -

PHP Warning:  include(phar:///home4/website/wp-cli.phar/vendor/composer/../wp-cli/php-cli-tools/lib/cli/Colors.php): failed to open stream: phar error: Cannot open phar archive "/home4/website/wp-cli.phar" for reading in phar:///home4/website/wp-cli.phar/vendor/composer/ClassLoader.php on line 444
Warning: include(phar:///home4/website/wp-cli.phar/vendor/composer/../wp-cli/php-cli-tools/lib/cli/Colors.php): failed to open stream: phar error: Cannot open phar archive "/home4/website/wp-cli.phar" for reading in phar:///home4/website/wp-cli.phar/vendor/composer/ClassLoader.php on line 444
PHP Warning:  include(): Failed opening 'phar:///home4/website/wp-cli.phar/vendor/composer/../wp-cli/php-cli-tools/lib/cli/Colors.php' for inclusion (include_path='.:/opt/cpanel/ea-php70/root/usr/share/pear') in phar:///home4/website/wp-cli.phar/vendor/composer/ClassLoader.php on line 444
Warning: include(): Failed opening 'phar:///home4/website/wp-cli.phar/vendor/composer/../wp-cli/php-cli-tools/lib/cli/Colors.php' for inclusion (include_path='.:/opt/cpanel/ea-php70/root/usr/share/pear') in phar:///home4/website/wp-cli.phar/vendor/composer/ClassLoader.php on line 444
PHP Warning:  require_once(phar://wp-cli.phar/vendor/wp-cli/php-cli-tools/lib/cli/Colors.php): failed to open stream: phar error: Cannot open phar archive "/home4/website/wp-cli.phar" for reading in phar:///home4/website/wp-cli.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Autoloader.php on line 128
Warning: require_once(phar://wp-cli.phar/vendor/wp-cli/php-cli-tools/lib/cli/Colors.php): failed to open stream: phar error: Cannot open phar archive "/home4/website/wp-cli.phar" for reading in phar:///home4/website/wp-cli.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Autoloader.php on line 128
PHP Fatal error:  require_once(): Failed opening required 'phar://wp-cli.phar/vendor/wp-cli/php-cli-tools/lib/cli/Colors.php' (include_path='.:/opt/cpanel/ea-php70/root/usr/share/pear') in phar:///home4/website/wp-cli.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Autoloader.php on line 128
Fatal error: require_once(): Failed opening required 'phar://wp-cli.phar/vendor/wp-cli/php-cli-tools/lib/cli/Colors.php' (include_path='.:/opt/cpanel/ea-php70/root/usr/share/pear') in phar:///home4/website/wp-cli.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Autoloader.php on line 128

I can't find many details about the specific errors being reported above, but I did come across https://github.com/pantheon-systems/terminus/issues/981 which talks about a similar issue (failed to open stream: phar error: Cannot open phar archive). It talks about the problem being related to a low value of ulimit

For reference, this is the output of the ulimit -a command on the server where I am testing -

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) 800000
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 235807
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 800000
open files                      (-n) 100
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 15240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 25
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[rmccue]

Oof, if you can't run wp-cli itself, that's going to be a problem.

However, since we're just using the S3 transfer facility under the hood, potentially you can run the AWS CLI directly instead. The command you'd want is aws s3 sync with something like:

aws s3 sync {dir} s3://bucket/{dir} --acl public-read

[rmccue]

(FWIW, it sounds like your ulimit may just be too low to be effective for running any command though; your host may be able to give you a temporary bump for running these commands.)