Every day : Server Crash

[Ribero]

v1.7.0 - Admin v1.10.0

Every morning when trying to access the site all users are stuck: Cannot access the grav instance which only displays instead of home page (admin page not accessible too):

Fatal error: Uncaught RuntimeException: Failed to save file /var/www/html/cache/compiled/blueprints/master-xx.xx.xx.xx.php in /var/www/html/vendor/rockettheme/toolbox/File/src/AbstractFile.php:369 
Stack trace: 
#0 /var/www/html/vendor/rockettheme/toolbox/File/src/PhpFile.php(41): RocketTheme\Toolbox\File\AbstractFile->save(Array) 
#1 /var/www/html/system/src/Grav/Common/Config/CompiledBase.php(255): RocketTheme\Toolbox\File\PhpFile->save(Ar
ray) 
#2 /var/www/html/system/src/Grav/Common/Config/CompiledBase.php(113): Grav\Common\Config\CompiledBase->saveCompiledFile('/var/www/html/c...') 
#3 /var/www/html/system/src/Grav/Common/Service/ConfigServiceProvider.php(92): Grav\Common\Config\CompiledBase->load() 
#4 /var/www/html/system/src/Grav/Common/Service/ConfigServiceProvider.php(45): Grav\Common\Service\ConfigServiceProvider::blueprints(Object(Grav\Common\Grav)) 
#5 /var/www/html/vendor/pimple/pimple/src/Pimple/Container.php(118): Grav\Common\Service\ConfigServiceProvider->Grav\Common\Service\{closure}(Object(Grav\Common\Grav)) 
#6 in /var/www/html/vendor/rockettheme/toolbox/File/src/AbstractFile.php on line 369

Forced to login in the docker container to do a php ./bin/grav cache --all

Everyday the Grav server crashes this way making it unusable. It is probably linked to a plugin (Presentation, Blue Prints ?) as the log is full of errors regarding blueprints (cf. #3159). I uninstalled all plugins and all themes and reinstalled the Themes after that but no change.

The scheduler is activated and launch cache-purge (4:00am), cache-clear (3:00am) and do an automated backup (2:00am) every day.

Logs: logs.zip

[mahagr]

Maybe your server changes the file permissions automatically? This error comes when files cannot be saved into the cache folder.

[Ribero]

well, the files & folder permissions looks ok except master-cli.php :

root@253d31ae1e89:/var/www/html/cache/compiled/blueprints# ls -lFa
total 1128
drwxr-xr-x. 2 www-data www-data    102 Jan 21 15:23 ./
drwxr-xr-x. 6 www-data www-data     68 Jan 21 15:15 ../
-rw-r--r--. 1 www-data www-data 398025 Jan 21 15:22 master-xxx.xxx.xxx.xxx.php
-rw-r--r--. 1 root     root     376519 Jan 21 15:23 master-cli.php
-rw-r--r--. 1 www-data www-data 376519 Jan 21 15:15 myserver.domain.php
root@253d31ae1e89:/var/www/html/cache/compiled/blueprints# 

I changed the owner of the file to reflect the other but just a few hours later I see that the file is once again owned by root. During this time I did not use command line or anything else. Only the Admin Panel Gui.

How to prevent this situation ?

[mahagr]

You really don't want to run CLI as root, like.. ever.

You should run it using the same user: sudo -u www-data bin/grav ... to keep the file permissions correct on all files.

The permission is correct now, but likely something changes those permissions during the night...?

[Ribero]

as stated I do not use any CLI as root... The file owner is changed by another process not by me !

[Ribero]

To illustrate:

chown -R www-data:www-data /var/www/html/    
ls -lFa /var/www/html/cache/compiled/blueprints/
total 736
drwxr-xr-x. 2 www-data www-data     58 Jan 21 16:53 ./
drwxr-xr-x. 6 www-data www-data     68 Jan 21 16:51 ../
-rw-r--r--. 1 www-data www-data 376519 Jan 21 16:51 master-xxx.xxx.xxx.xxx.php
-rw-r--r--. 1 www-data www-data 376519 Jan 21 16:53 master-cli.php
root@253d31ae1e89:/# 

Go to Admin Panel and clear cache (using icon top left) -> all files/folder are cleared:

root@253d31ae1e89:/# ls -lFa /var/www/html/cache/compiled/
total 4
drwxr-xr-x. 3 www-data www-data   19 Jan 21 16:51 ./
drwxrwxrwx. 7 www-data www-data  111 Jan 21 16:50 ../
drwxr-xr-x. 2 www-data www-data 4096 Jan 21 16:51 files/
root@253d31ae1e89:/var/www/html/cache/compiled#

then wait a few second for the cache/compilation to be reconstructed (takes 1min on my setup):

root@253d31ae1e89:/# ls -lFa /var/www/html/cache/compiled/blueprints/
total 760
drwxr-xr-x. 2 www-data www-data     58 Jan 21 16:52 ./
drwxr-xr-x. 6 www-data www-data     68 Jan 21 16:51 ../
-rw-r--r--. 1 www-data www-data 376519 Jan 21 16:51 master-34.91.56.93.php
-rw-r--r--. 1 root     root     398025 Jan 21 16:52 master-cli.php
root@253d31ae1e89:/# 

So my understanding is that the clear-cache feature called manually or through the scheduler effectively clear all caches but the re-compilation of some part of product are created using the wrong user ID.

[rhukster]

You should also make sure the scheduler command is in your users crontab and not in roots. If root runs the grav scheduler then the ownership will change to root. I think this is what you are seeing.

[Ribero]

Admin Panel clear cash is done using the scheduler ?

[rhukster]

No admin cache clear is done via php running under the web server. So whatever user runs httpd (or nginx etc)

[rhukster]

You mentioned scheduler and I’ve seen the situation here scheduler is running under another users crontab causing permission conflicts.

[Ribero]

My test case is not using the scheduler anyway I effectively have a crontab for user root that is launching the scheduler and this is certainly why I have the file master-cli.php owned by root.

I had to add the crontab to user root as the automatic backups were not launching with the webserver user.

I have now disabled the root crontab and need to investigate why backup are not running using web server user.

Thanks for your patience and guidance.

[Ribero]

I can confirm that I have no more issues with clear cache but I lost the ability to schedule backups