Stuck at "Starting a systemd service" for hours

[tio-trom]

Updating it from 4.3.1 ynh1 to ynh2 I see this for hours:

Now upgrading peertube...
Loading installation settings...
Checking version...
Backing up the app before upgrading (may take a while)...
Stopping a systemd service...
Ensuring downward compatibility...
NOTICE: extension "unaccent" already exists, skipping
NOTICE: extension "pg_trgm" already exists, skipping
'/etc/apt/sources.list.d/yarn.list' wasn't deleted because it doesn't exist.
'/usr/share/yunohost/hooks/conf_regen/15-nginx_peertube' wasn't deleted because it doesn't exist.
'/home/yunohost.app/peertube/storage/logs' wasn't deleted because it doesn't exist.
Making sure dedicated system user exists...
Upgrading dependencies...
Upgrading NGINX web server configuration...
Creating a data directory...
Building Yarn dependencies...
Updating a config file...
File /var/www/peertube/config/local-production.json has been manually modified since the installation or last upgrade. So it has been duplicated in /var/cache/yunohost/appconfbackup//var/www/peertube/config/local-production.json.backup.20221211.153248
Upgrading systemd configuration...
Starting a systemd service...
(this may take some time)

No other input. Opening the ynh admin dashboard in another tab prevents me from accessing the updates for instance, same as it does when a ynh process is running. Maybe it is a bug in the latest package version?

[tio-trom]

Eventually it worked....

[fflorent]

I propose to reopen the issue, I have the same problem and it's a quite annoying one.

[yalh76]

As far as i've seen, usually it's because the service starts too quickly

[madmaxlamenace]

I have the same issue Info : Mise à jour de peertube... Info : [....................] > Loading installation settings... Info : [+...................] > Checking version... Info : [#+..................] > Backing up the app before upgrading (may take a while)... Attention : $NO_BACKUP_UPGRADE is set, backup will be avoided. Be careful, this upgrade is going to be operated without a security backup Info : [##+.................] > Stopping a systemd service... Info : [###.................] > Ensuring downward compatibility... Attention : NOTICE: l'extension « unaccent » existe déjà, poursuite du traitement Attention : NOTICE: l'extension « pg_trgm » existe déjà, poursuite du traitement Info : '/etc/apt/sources.list.d/yarn.list' wasn't deleted because it doesn't exist. Info : '/usr/share/yunohost/hooks/conf_regen/15-nginx_peertube' wasn't deleted because it doesn't exist. Info : '/home/yunohost.app/peertube/storage/logs' wasn't deleted because it doesn't exist. Info : [###+................] > Making sure dedicated system user exists... Info : [####+...............] > Upgrading source files... Info : [#####+..............] > Upgrading dependencies... Info : [######..............] > Upgrading NGINX web server configuration... Info : [######+.............] > Creating a data directory... Info : [#######+............] > Building Yarn dependencies... Info : [########+...........] > Updating a config file... Info : [#########+..........] > Upgrading systemd configuration... Info : [##########..........] > Starting a systemd service... Attention : (this may take some time)

[TILvids]

I think this is the same thing I'm running into here https://github.com/YunoHost-Apps/peertube_ynh/issues/365

I'll close that issue so we can focus on it here. Definitely I was hung up on "Starting a systemd service" for many hours before eventually it failed out and PeerTube would no longer load up.

[yalh76]

You should look at :

• journalctl -feu peertube
• tail -F /var/log/peertube/peertube.log to see what's happen

[julianfoad]

I think I found the source of the problem. (I am upgrading from peertube 5.1 to 5.2.)

The upgrade script at line 261 is stuck, watching the peertube log file /var/log/peertube/peertube.log for a line matching HTTP server listening on localhost, with a timeout of 5400 seconds (1.5 hours):

ynh_systemd_action --service_name=$app --action="start" --log_path="/var/log/$app/$app.log" --line_match="HTTP server listening on localhost" --timeout=5400 --length=200

What actually happens is peertube is logging into the file /var/log/peertube/peertube5.log (in my case), and so the upgrade script never sees the line where it is watching. See how peertube created multiple log files over time:

$ sudo ls -laF /var/log/peertube/
total 66552
drwxrwxrwx  2 peertube peertube     4096 Jun 10 09:38 ./
drwxr-xr-x 23 root     root         4096 Jun 21 00:00 ../
-rw-r--r--  1 peertube peertube 12595469 Apr 11 11:00 peertube1.log
-rw-r--r--  1 peertube peertube 12583200 Apr  1 13:00 peertube2.log
-rw-r--r--  1 peertube peertube 12596028 Jun 21 20:38 peertube3.log
-rw-r--r--  1 peertube peertube 12584637 Jun 10 09:38 peertube4.log
-rw-r--r--  1 peertube peertube  5012474 Jun 23 10:13 peertube5.log
-rw-r--r--  1 peertube peertube   134259 Jun 22 08:42 peertube-audit.log
-rw-r--r--  1 peertube peertube 12587726 Feb  4 20:58 peertube.log

(I notice that these are not even in date order.)

The upgrade does indeed write the expected line, just not in the expected file:

$ grep "HTTP server listening on localhost:" /var/log/peertube/peertube5.log
[...some previous log entries match too...]
{"level":"info","message":"HTTP server listening on localhost:8095","label":"example.org:443","timestamp":"2023-06-23T10:05:50.145Z"}

Implications

The implication is this bug is going to show up only if the peertube installation has been in place long enough for it to have multiple log files. The log files seem to be rotated (new name chosen) once they reach about 10 MB in size, from my observation, which in my case is around two months.

A temporary work-around

When I manually add the expected line to peertube.log like this:

$ echo '{"level":"info","message":"HTTP server listening on localhost:8095","label":"example.org:443","timestamp":"2023-06-23T09:09:46.721Z"}' | sudo tee -a /var/log/peertube/peertube.log

then the installation continues, and completes.

(For this work-around I appended a whole, correctly formatted log entry, rather than just the text the script is watching for, in order not to risk breaking the yunohost log viewer or other functions that might possibly try to parse this log file later.)

Using this work-around before the 1.5-hour timeout, I successfully completed the upgrade on two separate installations.

---

POSSE

[TILvids]

I might give this a try, thank you for posting. I still have not been able to upgrade our instance.

[julianfoad]

@TILvids I forgot to say how I detected it was in this state. Can't remember now but I think I must have seen the ynh_systemd_action command in ps or in top. Please do report back with what you find.