search

jarischaefer / docker-librenms

Docker image for LibreNMS
MIT License
115 stars 37 forks source link

ProcessTimedOutException #136

Closed Kia0ra closed 10 months ago

Kia0ra commented 2 years ago

Hi,

I use your container on a Synology NAS (Intel), I had not updated it for a while (maybe a year or more), since I update to last version (22.4.1) I can't launch the container, something to do with PHP7.4 ?

*** Running /etc/my_init.d/librenms_010_mounts...
*** Running /etc/my_init.d/librenms_100_cron...
*** Running /etc/my_init.d/librenms_101_ssl...
*** Running /etc/my_init.d/librenms_102_ipv6...
*** Running /etc/my_init.d/librenms_103_timezone...
*** Running /etc/my_init.d/librenms_104_config...
> Illuminate\Foundation\ComposerScripts::postInstall
> LibreNMS\ComposerHelper::postInstall
> Illuminate\Foundation\ComposerScripts::postInstall
> @php artisan vue-i18n:generate --multi-locales --format=umd

  [Symfony\Component\Process\Exception\ProcessTimedOutException]                                                                                                                                   
  The process "'/usr/bin/php7.4' -d allow_url_fopen='1' -d disable_functions='' -d memory_limit='-1' artisan vue-i18n:generate --multi-locales --format=umd" exceeded the timeout of 300 seconds.  

run-script [--timeout TIMEOUT] [--dev] [--no-dev] [-l|--list] [--] [<script>] [<args>]...

*** /etc/my_init.d/librenms_104_config failed with status 1

*** Killing all processes...
May 19 06:12:52 librenms syslog-ng[17]: syslog-ng shutting down; version='3.25.1'

Thanks

jarischaefer commented 2 years ago

@Kia0ra Try pulling the latest build, hopefully the larger timeout (3600s instead of 300s) will work.

Kia0ra commented 2 years ago

Thank you @jarischaefer for the new release. I have tried, it helped, the container went further in the startup but unfortunately it remains unworkable.

It took almost 2 hours (!) to run all the "php artisan" scripts but although the nginx process was started at the end, it remained unreachable (504 - Gateway Timeout), the container was saturated with PHP requests from CRON and couldn't keep up, it ended up using 1.5GB of ram out of the 10GB and I stopped it.

Jun  1 09:43:08 librenms syslog-ng[17]: syslog-ng starting up; version='3.25.1'
*** Running /etc/my_init.d/librenms_000_permissions...
setfacl: /opt/librenms/bootstrap/cache: Operation not supported
setfacl: /opt/librenms/logs: Operation not supported
setfacl: /opt/librenms/rrd: Operation not supported
setfacl: /opt/librenms/storage: Operation not supported
*** Running /etc/my_init.d/librenms_010_mounts...
*** Running /etc/my_init.d/librenms_100_cron...
*** Running /etc/my_init.d/librenms_101_ssl...
*** Running /etc/my_init.d/librenms_102_ipv6...
*** Running /etc/my_init.d/librenms_103_timezone...
*** Running /etc/my_init.d/librenms_104_config...
> Illuminate\Foundation\ComposerScripts::postInstall
> LibreNMS\ComposerHelper::postInstall
> Illuminate\Foundation\ComposerScripts::postInstall
> @php artisan vue-i18n:generate --multi-locales --format=umd
> @php artisan view:cache
Compiled views cleared!
Blade templates cached successfully!
> @php artisan optimize
Configuration cache cleared!
Configuration cached successfully!
Route cache cleared!
Routes cached successfully!
Files cached successfully!
> @php artisan config:clear
Configuration cache cleared!
> scripts/dynamic_check_requirements.py || pip3 install --user -r requirements.txt || :
*** Running /etc/my_init.d/librenms_105_environment...
*** Running /etc/my_init.d/librenms_106_daily...
*** Running /etc/my_init.d/librenms_107_create_default_files...
*** Running /etc/my_init.d/librenms_108_syslog...
*** Running /etc/my_init.d/librenms_109_services...
*** Running /etc/my_init.d/librenms_900_permissions...
setfacl: /opt/librenms/bootstrap/cache: Operation not supported
setfacl: /opt/librenms/logs: Operation not supported
setfacl: /opt/librenms/rrd: Operation not supported
setfacl: /opt/librenms/rrd/192.168.0.252: Operation not supported
setfacl: /opt/librenms/rrd/192.168.0.1: Operation not supported
setfacl: /opt/librenms/storage: Operation not supported
setfacl: /opt/librenms/storage/app: Operation not supported
setfacl: /opt/librenms/storage/app/public: Operation not supported
setfacl: /opt/librenms/storage/app/purifier: Operation not supported
setfacl: /opt/librenms/storage/app/purifier/HTML: Operation not supported
setfacl: /opt/librenms/storage/app/purifier/URI: Operation not supported
setfacl: /opt/librenms/storage/debugbar: Operation not supported
setfacl: /opt/librenms/storage/framework: Operation not supported
setfacl: /opt/librenms/storage/framework/cache: Operation not supported
setfacl: /opt/librenms/storage/framework/cache/data: Operation not supported
setfacl: /opt/librenms/storage/framework/sessions: Operation not supported
setfacl: /opt/librenms/storage/framework/testing: Operation not supported
setfacl: /opt/librenms/storage/framework/views: Operation not supported
setfacl: /opt/librenms/storage/logs: Operation not supported
*** Booting runit daemon...
*** Runit started as PID 231
Starting rrdcached...
Starting php-fpm...
Starting nginx...
Jun  1 12:15:36 librenms cron[240]: (CRON) INFO (pidfile fd = 3)
Jun  1 12:15:36 librenms cron[240]: (CRON) INFO (Running @reboot jobs)
Jun  1 12:16:01 librenms CRON[258]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:17:01 librenms CRON[263]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:17:01 librenms CRON[266]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jun  1 12:18:01 librenms CRON[271]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:19:01 librenms CRON[275]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:20:01 librenms CRON[284]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/lib/sm.bin/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun  1 12:20:01 librenms CRON[287]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/poller-wrapper.py 8 >> /dev/null 2>&1)
Jun  1 12:20:01 librenms CRON[288]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jun  1 12:20:01 librenms CRON[290]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/check-services.php >> /dev/null 2>&1)
Jun  1 12:20:01 librenms CRON[286]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:20:01 librenms CRON[297]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jun  1 12:21:01 librenms CRON[323]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:22:01 librenms CRON[331]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:23:01 librenms CRON[342]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:24:01 librenms CRON[359]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:25:01 librenms CRON[370]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/check-services.php >> /dev/null 2>&1)
Jun  1 12:25:01 librenms CRON[372]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:25:01 librenms CRON[371]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jun  1 12:25:01 librenms CRON[373]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/poller-wrapper.py 8 >> /dev/null 2>&1)
Jun  1 12:25:02 librenms CRON[380]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jun  1 12:26:01 librenms CRON[399]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:27:01 librenms CRON[432]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:28:01 librenms CRON[801]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:29:01 librenms CRON[844]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:30:01 librenms CRON[867]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/poller-wrapper.py 8 >> /dev/null 2>&1)
Jun  1 12:30:01 librenms CRON[868]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jun  1 12:30:01 librenms CRON[873]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:30:01 librenms CRON[874]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jun  1 12:30:01 librenms CRON[877]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/check-services.php >> /dev/null 2>&1)
Jun  1 12:31:01 librenms CRON[921]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:32:01 librenms CRON[950]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:33:01 librenms CRON[989]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:33:01 librenms CRON[992]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/discovery-wrapper.py 1 >> /dev/null 2>&1)
Jun  1 12:34:01 librenms CRON[1028]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:35:01 librenms CRON[1078]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/poller-wrapper.py 8 >> /dev/null 2>&1)
Jun  1 12:35:01 librenms CRON[1079]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jun  1 12:35:01 librenms CRON[1077]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:35:01 librenms CRON[1075]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/check-services.php >> /dev/null 2>&1)
Jun  1 12:35:01 librenms CRON[1087]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jun  1 12:36:01 librenms CRON[1132]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:37:01 librenms CRON[1185]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:38:02 librenms CRON[1228]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:39:01 librenms CRON[1287]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:39:01 librenms CRON[1290]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jun  1 12:40:01 librenms CRON[1376]: (smmsp) CMD (test -x /etc/init.d/sendmail && test -x /usr/share/sendmail/sendmail && test -x /usr/lib/sm.bin/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun  1 12:40:01 librenms CRON[1378]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:40:01 librenms CRON[1379]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jun  1 12:40:01 librenms CRON[1380]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/check-services.php >> /dev/null 2>&1)
Jun  1 12:40:01 librenms CRON[1386]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/poller-wrapper.py 8 >> /dev/null 2>&1)
Jun  1 12:40:01 librenms CRON[1389]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jun  1 12:41:01 librenms CRON[1461]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:42:03 librenms CRON[1511]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:43:01 librenms CRON[1540]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:44:01 librenms CRON[1586]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:45:01 librenms CRON[1628]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/cronic /opt/librenms/poller-wrapper.py 8 >> /dev/null 2>&1)
Jun  1 12:45:01 librenms CRON[1629]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/discovery.php -h new >> /dev/null 2>&1)
Jun  1 12:45:01 librenms CRON[1630]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
Jun  1 12:45:01 librenms CRON[1636]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/poll-billing.php >> /dev/null 2>&1)
Jun  1 12:45:01 librenms CRON[1638]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/check-services.php >> /dev/null 2>&1)
Jun  1 12:46:01 librenms CRON[1687]: (librenms) CMD (. /etc/librenms_environment; /opt/librenms/alerts.php >> /dev/null 2>&1)
*** Shutting down runit daemon (PID 231)...
*** Init system aborted.
*** Killing all processes...
Jun  1 12:46:20 librenms syslog-ng[17]: syslog-ng shutting down; version='3.25.1'
jarischaefer commented 2 years ago

@Kia0ra Is it possible that the underlying storage is slow, the CPU is busy or memory pressure is great? Perhaps you could try running a fresh instance using the beta build 22.6.0-ubuntu2204-b1:

  1. Download https://raw.githubusercontent.com/jarischaefer/docker-librenms/ubuntu-22.04/docker-compose.yml
  2. Replace jarischaefer/docker-librenms with jarischaefer/docker-librenms:22.6.0-ubuntu2204-b1
  3. Run docker-compose up in the directory which contains docker-compose.yml
  4. MySQL should come up after a while
  5. Once MySQL is healthy, LibreNMS should come up

You may gather information on resource usage using commands such as top, htop, iotop, iostat 1, vmstat 1 or smem --sort=swap.