Too many open files on Siteground

[andrewspringman]

I'm getting a 500 error when trying to create a collection on SiteGround

The php_errorlog shows

[21-Apr-2018 12:25:46 UTC] PHP Fatal error:  Uncaught exception 'RuntimeException' with message 'SplFileObject::__construct(/home/[redacted]/public_html/cockpit-master/assets/app/media/icons/image.svg): failed to open stream: Too many open files' in /home/[redacted]/public_html/cockpit-master/lib/Lime/Helper/Filesystem.php:48
Stack trace:
#0 /home/[redacted]/public_html/cockpit-master/lib/Lime/Helper/Filesystem.php(48): SplFileObject->__construct('/home/[redacted]/...')
#1 /home/[redacted]/public_html/cockpit-master/storage/tmp/collection.php.b2f467b25b0eee2f9e5a113d790beecc.lexy.php(39): Lime\Helper\Filesystem->ls('*.svg', 'assets:app/medi...')
#2 /home/[redacted]/public_html/cockpit-master/lib/Lexy.php(435): include('/home/[redacted]/...')
#3 /home/[redacted]/public_html/cockpit-master/lib/Lexy.php(124): lexy_include_with_params('/home/[redacted]/...', Array, '/home/[redacted]/...')
#4 /home/[redacted]/public_html/cockpit-master/lib/LimeExtra/App.php(107): Lexy->file('/home/[redacted]/...', Array)
#5 /home/[redacted]/public_html/cockpit-master/lib/LimeExtra/Controller.php(43): in /home/[redacted]/public_html/cockpit-master/lib/Lime/Helper/Filesystem.php on line 48

When I ssh into siteground "ulimit -n" shows 1024. It's shared hosting so it won't let me increase the limit and Siteground won't do it for me.

I'm not running any other processes. There is no traffic to speak of on the site yet. Everything works fine in my local Docker container. It fails with both Master and Next on SiteGround I've tried PHP 7.2 (which fails for a different reason), PHP 7.1 and PHP 5.6. This is a new install, so there shouldn't be many files yet. storage and all subdirectories are writable. files are being written to storage/tmp. assets/app/media/icons has 60 files in it. Nowhere near 1024.

Is there a way around this beside upgrading my hosting account or switching providers? Perhaps a code patch?

If not, anybody successfully using Cockpit CMS on Siteground, what service level do I need?

[andrewspringman]

I got word from Siteground that all of the shared hosting tiers have ulimit -n == '1024'. The first non-shared tier is cloud hosting. So, if there isn't a code solution, it would be way cheaper to just host a docker container on hyper.sh. ~$3/mo vs ~$80/mo.

BTW, Directus works on Siteground, but I think Cockpit is a better solution for me as the developer (Directus doesn't have anything equivalent to Cockpit's Regions) and the backend makes more sense for my content editors (although not as pretty).

[aheinze]

this is the first time that this issue is mentioned 🤔 on which view do get that error?

[andrewspringman]

From the Dashboard clicking on "Create a collection" OR From Collections clicking on "Create one"

[Ghadyk]

I also have the same issue for Collections, Forms and Regions. Any updates on this?

[waywardm]

mine has the same error anyone any ideas hoe to fix this?

[raffaelj]

Same problem, but not on Siteground. I tried Cockpit on a shared host from strato. It's a new and empty installation and I had to change some paths via defines.php (Steps to reproduce: https://github.com/agentejo/cockpit/issues/704#issuecomment-405016371). When I try to create a new collection, I got a 500 error.

After activating debug, I got

Uncaught RuntimeException: SplFileObject::__construct(/path/to/htdocs/redesign/admin/assets/app/media/icons/text.svg): failed to open stream: Too many open files in /path/to/htdocs/redesign/admin/lib/Lime/Helper/Filesystem.php:50
Stack trace:
#0 /path/to/htdocs/redesign/admin/lib/Lime/Helper/Filesystem.php(50): SplFileObject->__construct('/path/to/...')
#1 /path/to/htdocs/redesign/admin/storage/tmp/collection.php.9022a9d0e450112f73f15d8c433a4b21.lexy.php(39): Lime\Helper\Filesystem->ls('*.svg', 'assets:app/medi...')
#2 /path/to/htdocs/redesign/admin/lib/Lexy.php(435): include('/path/to/...')
#3 /path/to/htdocs/redesign/admin/lib/Lexy.php(124): lexy_include_with_params('/path/to/...', Array, '/path/to/...')
#4 /path/to/htdocs/redesign/admin/lib/LimeExtra/App.php(107): Lexy->file('/path/to/...', Array)
#5 /path/to/htdocs/redesign/admin/lib/

Trying to create region/singleton/form, it fails at .../icons/tickets.svg.

I than tried it with disabled PHP Boost/FastCGI (Stackoverflow, different problem) and I got nearly the same error. But this time it failed at .../icons/settings.svg instead of .../icons/text.svg.

I don't have a lot control via SSH there (ulimit: Command not found.).

I found a notice at php.net - comment:

Don't store DirectoryIterator objects for later; you will get an error saying "too many open files" when you store more than the operating system limit (usually 256 or 1024).

[raffaelj]

The problem seems to be in the Lime/Helper Filesystem.php: https://github.com/agentejo/cockpit/blob/next/lib/Lime/Helper/Filesystem.php#L50

The function ls() creates a lot of new SplFileObjects, which might not be neccessary. When the collection view tries to load all svg files in the icon folder, the server reaches the limit for opened files.

Here is a dirty hack to avoid creating new SplFileObjects in the views of collection, region, forms and singletons: https://github.com/raffaelj/cockpit/commit/b6882b6a87be91ac08667841b196a45b5f9ec3f9

I didn't do a lot of tests, but it seems to work.

After writing the workaround I wondered, why the normal dashboard works. The call of all these svg files is managed very different in ...layouts/app.php and without calling $app->helper("fs")->ls(...): https://github.com/agentejo/cockpit/blob/next/modules/Cockpit/views/layouts/app.php#L3 https://github.com/agentejo/cockpit/blob/next/modules/Cockpit/views/layouts/app.php#L103

I don't understand everything here, but it feels like I found the place(s) to look for. edit: --> OK, I got it. I didn't realize, that I can choose between 59 icons for a collection. This is the place, where all other pages don't need to load the whole icon folder. I thougt, the icons are loaded in the background for navigation etc.

@aheinze What's your opinion and could you point me in the right direction, please? It is a project for a client and I don't want to change the webhoster just for cockpit

[waywardm]

I will try and see if it works on mine, I have had to resort to using drupal installation as a headless cms for the time being. I’m slightly nervous of using a software that could easily break or not be supported further down the line as I don’t have hardly any programming knowledge. However a simple cms solution that could be placed on any shared hosting server with relative ease would be ideal.

[waywardm]

@raffaelj your fix definitely works on mine, definitely worth submitting the pull request as it may mean the code is working on shared hosting in other future code updates

[raffaelj]

@waywardm I want to wait for feedback from @aheinze because:

I solved the problem for this usecase only. The issue could/will occur again in other places. I did a test right now with my fix and created 50+ new collections until I got the next Uncaught RuntimeException: SplFileObject.... This time function ls creates new SplFileObjects for each collection.

I create a new issue, since this behavior could break Cockpit even on hosts with less limitations in different scenarios.

[aheinze]

@raffaelj it is important to have a more generic solution here. It seems that a SplFileObject keeps the filehandler open, which causes the error in some circumstances (I didn't experience the issue yet).

I think a solution could be a "dummy" CustomFileObject which just stores the file path internally and emulates the methods if needed. 🤔

[aheinze]

@raffaelj @waywardm could you please test the latest next code base and report if the error still occurs?

[raffaelj]

It works. Thank you very much.

I was aware of needing a more robust solution and I had the idea with a dummy class, too. This is why I called it a "dirty hack". But I would have to do a lot of research about file handling for it. Thanks for doing the work.

I tested it on strato with unknown ulimit and on an Uberspace7 with ulimit 1024. Both are able to create new collections.

I wasn't able to update the folders on strato (wget ..., cp -u -r ...) with existing config and storage for any reason (404 on collections page). But I can't reproduce it. I did too much tests and rewrote some code serverside...

A fresh installation (wget https://github.com/agentejo/cockpit/archive/next.zip) worked fine on both servers. Updating an installation with existing storage on Uberspace via git pull worked, too.

Now I'm happy :smile:

[aheinze]

@raffaelj thanks for testing 👍

[waywardm]

@aheinze Absolutely Excellent @raffaelj the latest code base on a fresh installation on a Siteground shared hosting works as expected so I am looking forward to trying out properly with Gatsbyjs, many thanks to both of you for looking into this issue. Initial tests with gatsby are far cleaner on api side than using headless drupal etc