Friendica worker queue is clocked/doesn't get executed

aarontrom commented 1 year ago

Describe the bug

The problem is, that the worker queue is not getting triggered/executed anymore... It used to work perfectly fine, but now systemctl status says the following:

var/www/friendica# systemctl status friendica-daemon.service 
● friendica-daemon.service - Friendica daemon
     Loaded: loaded (/etc/systemd/system/friendica-daemon.service; disabled; vendor preset: enabled)
     Active: failed (Result: start-limit-hit) since Wed 2023-06-21 00:38:01 CEST; 5 days ago
   Main PID: 1516554 (code=exited, status=0/SUCCESS)
        CPU: 259ms

Jun 21 00:38:01 tromdienste.de systemd[1]: friendica-daemon.service: Scheduled restart job, restart counter is at 15.
Jun 21 00:38:01 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 21 00:38:01 tromdienste.de systemd[1]: friendica-daemon.service: Start request repeated too quickly.
Jun 21 00:38:01 tromdienste.de systemd[1]: friendica-daemon.service: Failed with result 'start-limit-hit'.
Jun 21 00:38:01 tromdienste.de systemd[1]: Failed to start Friendica daemon.
Jun 21 02:53:18 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:11: Failed to parse output specifier, ignoring: journalctl

The admin-panel in friendica says the following things:

There is a new version of Friendica. You are currently using version 2023.01, the current version is 2023.09-dev.

The background process (worker) was last executed at 2023-06-20 09:53:39 UTC. That was more than an hour ago. Please check your crontab settings.

The log file 'log/friendica.log' is not writable. Currently no logging is possible (Error: 'Directory "" cannot get created: Invalid path').

Context

Hardware: VPS bought online
YunoHost version: 11.1.21.4 (stable)
I have access to my server: Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no
- If yes, please explain:
Using friendica package version/branch: Friendica 'Giant Rhubarb' 2023.01 - 1502
Version Friendica 'Giant Rhubarb' 2023.01 - 1502

Steps to reproduce

I installed friendica a while ago and did set up the friendica daemon via the following tutorial: https://nequalsonelifestyle.com/2022/08/04/setting-up-friendica-daemon-systemd-service/

then I also did set up the cronjob as described in the tutorial https://wiki.friendi.ca/docs/install: /10 * cd /var/www/friendica; /usr/bin/php bin/worker.php

the current friendica-daemon.service is configured as such:

[Unit]
Description=Friendica daemon
After=network.target mariadb.service
Requires=network.target remote-fs.target nss-lookup.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/friendica
Type=simple
StandardError=journalctl
ExecStart=/usr/bin/php ./bin/daemon.php start
ExecStop=/usr/bin/php ./bin/daemon.php stop
PIDFile=friendica/daemon.pid
PrivateTmp=true
InaccessibleDirectories=/home /root /boot /opt /mnt /media
ReadOnlyDirectories=/etc /usr
Restart=always
StartLimitBurst=15
# Restart but not more than once every 10 minutes 
# StartLimitInteraval=600
# Restart, but not more than once every 30s (for testing puropses)
StartLimitInterval=30

[Install]
WantedBy=multi-user.target

Expected behavior

The worker queue to be executed.

Logs

I don't really know why the friendica daemon stopped working. What I found in the logs is the following:

May 13 13:02:53 tromdienste.de php[2643603]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 14 01:12:12 tromdienste.de systemd[1]: Stopping Friendica daemon...
May 14 01:12:14 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
May 14 01:12:14 tromdienste.de systemd[1]: Stopped Friendica daemon.
May 14 01:12:14 tromdienste.de systemd[1]: friendica-daemon.service: Consumed 1d 3h 8min 56.504s CPU time.
-- Boot 18ee3c68146b441c8df818d697379963 --
May 14 01:16:45 tromdienste.de systemd[1]: Started Friendica daemon.
May 16 03:29:20 tromdienste.de php[962970]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 16 13:06:06 tromdienste.de php[1124165]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 18 21:23:16 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
May 19 21:53:17 tromdienste.de php[2801409]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 22 22:26:54 tromdienste.de php[175712]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 24 21:00:02 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
May 24 21:00:03 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
May 27 07:41:31 tromdienste.de php[2257406]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 27 07:58:56 tromdienste.de php[2262327]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
May 27 14:46:47 tromdienste.de systemd[1]: Stopping Friendica daemon...
May 27 14:46:48 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
May 27 14:46:48 tromdienste.de systemd[1]: Stopped Friendica daemon.
May 27 14:46:48 tromdienste.de systemd[1]: friendica-daemon.service: Consumed 1d 21h 52min 45.016s CPU time.
-- Boot 7d800d72e1134fbab2b2876895be51d3 --
May 27 14:48:08 tromdienste.de systemd[1]: Started Friendica daemon.
May 30 01:01:30 tromdienste.de php[1175059]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96
Jun 02 01:30:40 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 02 01:30:56 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 02 01:33:22 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 02 01:33:58 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 02 01:34:00 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 02 01:34:00 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 02 14:55:11 tromdienste.de php[2855047]: PHP Warning:  Invalid argument supplied for foreach() in /var/www/friendica/src/Worker/UpdateServerDirectory.php on line 96

and here is where the worker queue stopped getting executed:

Jun 07 00:06:48 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 07 00:06:50 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 07 00:06:50 tromdienste.de systemd[1]: /etc/systemd/system/friendica-daemon.service:12: Standard output type syslog is obsolete, automatically updating to journal. Please update yo>
Jun 07 02:15:56 tromdienste.de systemd[1]: Stopping Friendica daemon...
Jun 07 02:15:57 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
Jun 07 02:15:57 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 07 02:15:57 tromdienste.de systemd[1]: friendica-daemon.service: Consumed 3h 9min 35.189s CPU time.
-- Boot 0f50635c110943f8a8831f6c7300c3c1 --
Jun 07 02:17:30 tromdienste.de systemd[1]: Started Friendica daemon.
Jun 07 02:17:32 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
Jun 07 02:17:32 tromdienste.de systemd[1]: friendica-daemon.service: Consumed 1.012s CPU time.
Jun 07 02:17:33 tromdienste.de systemd[1]: friendica-daemon.service: Scheduled restart job, restart counter is at 1.
Jun 07 02:17:33 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 07 02:17:33 tromdienste.de systemd[1]: friendica-daemon.service: Consumed 1.012s CPU time.
Jun 07 02:17:33 tromdienste.de systemd[1]: Started Friendica daemon.
Jun 07 02:17:33 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
Jun 07 02:17:33 tromdienste.de systemd[1]: friendica-daemon.service: Scheduled restart job, restart counter is at 2.
Jun 07 02:17:33 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 07 02:17:33 tromdienste.de systemd[1]: Started Friendica daemon.
Jun 07 02:17:34 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
Jun 07 02:17:34 tromdienste.de systemd[1]: friendica-daemon.service: Scheduled restart job, restart counter is at 3.
Jun 07 02:17:34 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 07 02:17:34 tromdienste.de systemd[1]: Started Friendica daemon.
Jun 07 02:17:34 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
Jun 07 02:17:35 tromdienste.de systemd[1]: friendica-daemon.service: Scheduled restart job, restart counter is at 4.
Jun 07 02:17:35 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 07 02:17:35 tromdienste.de systemd[1]: Started Friendica daemon.
Jun 07 02:17:35 tromdienste.de systemd[1]: friendica-daemon.service: Succeeded.
Jun 07 02:17:35 tromdienste.de systemd[1]: friendica-daemon.service: Scheduled restart job, restart counter is at 5.
Jun 07 02:17:35 tromdienste.de systemd[1]: Stopped Friendica daemon.
Jun 07 02:17:35 tromdienste.de systemd[1]: friendica-daemon.service: Start request repeated too quickly.
Jun 07 02:17:35 tromdienste.de systemd[1]: friendica-daemon.service: Failed with result 'start-limit-hit'.
Jun 07 02:17:35 tromdienste.de systemd[1]: Failed to start Friendica daemon.

If anyone can help, I would be so glad, because my posts are not getting sent out at the moment I think.

aarontrom commented 1 year ago

at the moment I have the following issue:

/var/www/friendica# sudo -u www-data bin/daemon.php start
PHP Fatal error:  Uncaught Error: Cannot instantiate interface Friendica\Core\Addon\Capability\ICanLoadAddons in /var/www/friendica/vendor/level-2/dice/Dice.php:130
Stack trace:
#0 /var/www/friendica/vendor/level-2/dice/Dice.php(130): ReflectionClass->newInstanceWithoutConstructor()
#1 /var/www/friendica/vendor/level-2/dice/Dice.php(96): Dice\Dice->Dice\{closure}()
#2 /var/www/friendica/bin/daemon.php(64): Dice\Dice->create()
#3 {main}
  thrown in /var/www/friendica/vendor/level-2/dice/Dice.php on line 130

MrPetovan commented 1 year ago

Thanks for the last bit of context, this is what we needed to fix the issue.

MrPetovan commented 1 year ago

It looks like you're using the develop version of bin/daemon.php with the stable version of static/dependencies.config.php.

The Friendica\Core\Addon\Capability\ICanLoadAddons interface is only defined in develop.

I checked the master version of the Friendica YunoHost App manifest, it does reference the correct tag 2023.05 which has both files in a matching version, so I'm confused as of why you would have unmatching versions on your install.

aarontrom commented 1 year ago

Ohh okay, I think I'm not sure if I understand that correctly. What does that mean exactly?

In the meantime I also upgraded Friendica to the version 'Giant Rhubarb' 2023.05 - 1518, but that issue is still there. Can or should I do something about that?

MrPetovan commented 1 year ago

This means:

your static/dependencies.config.php file is correctly in the 2023.05 version.
your bin/daemon.php file is incorrectly in a newer version than 2023.05.

This version mismatch is triggering the error you've reported. Since you're relying on YunoHost to deploy files, I'm not sure what's the extent of the version mismatch throughout your Friendica file tree, but the GitHub tag archive YunoHost is supposed to use looks okay.

aarontrom commented 1 year ago

Okay thanks a lot Hypolite!

aarontrom commented 1 year ago

I'm thinking about backing up the friendica database, then reinstalling friendica and then importing the database again - I tried that already, but I'm not sure if I tried that correctly, as I couldn't log in anymore... is there anything I would need to pay attention to? Or do you think that can't work at all?

MrPetovan commented 1 year ago

This is actually a great idea, instead of going through the upgrade process on an ancient install.

aarontrom commented 1 year ago

Unfortunately, that didn't work. Maybe I did something wrong, but I followed this tutorial: https://www.sqlshack.com/how-to-backup-and-restore-mysql-databases-using-the-mysqldump-command/ maybe I try 1 more time later on, but this time, friendica had no data at all.

tio-trom commented 3 months ago

@aarontrom you still have this issue? I suppose you dont run Friendica anymore? Should this issue be closed?

YunoHost-Apps / friendica_ynh