Documentation: Semantics of timer expiry isn't well documented

[nh2]

The systemd.timer man page currently does not describe important aspects of timer semantics, which can lead to unexpected/unintended/wrong behaviour when used and be very frustrating.

I'll first show an example problem, and then ask some concrete questions that I couldn't answer from the current documentation.

Example problem

When I systemctl stop a.service, I often get the output Job for a.service canceled.

After some investigation and lots of great help from boucman on IRC, we tracked it down to a timer b.timer + b.service, where b.service is Requires=/After= on a.service.

The timer is set to OnUnitActiveSec=1m; the started service itself can take longer than a minute to complete its task. We observe that:

• systemctl stop a.service stops a
• a is then immediately restarted, because the b.timer immediately triggers, and the dependency pulls in a
• that seems to result in the Job for a.service canceled message.

When investigating whether this is as expected, the problem here is that nobody seems to really understand when timers fire, and that the documentation leaves these things open. Thus:

Things to be documented

• When and how do timers fire, concretely?
  • After I stop the service associated with a OnUnitActiveSec= timer, should the timer fire immediately, or shold it fire after some time (so that the difference between the previous start time and the new start time is divisible by 1m)?
  • The docs say "OnUnitActiveSec= defines a timer relative to when the unit the timer is activating was last activated" but don't describe how this interacts with stopping a service.
  • Concrete example: Timer is set to OnUnitActiveSec=1m. I start the service at time t=0s. I systemctl stop the service at t=90s. Should the service next be started (a) immediately, or (b) at t=120s?
  • As per my findings above, (a) is the case, but why, and what is the rationale behind it? Does systemd somehow remember that a timer would have fired? I would like to document it.
  • Note in the above, we can assume AccuracySec=1s or something similarly small so that this doesn't create extra confusion.
• Does it make sense that a timer expiring immediately this way gets bundled into the same transaction as the systemctl stop a.service, so that the result can only be Job for a.service canceled? If yes, how should this be explained in the documentation?

After these questions are cleared, we'd like to document these semantics in the systemd.timers man page.

[nh2]

CC @arianvp

[nh2]

I forgot to say: The Job for a.service canceled happens most of the time, but not all the time, so there seems to be some race in it.

[nh2]

Putting in some more details as asked on IRC:

• Both a.service and b.service don't take long to stop (they react in < 1s to a SIGTERM, and exit 1 when it occurs)

[flokli]

This is effectively preventing switching configuration / restarting reliably.

Some context: https://github.com/NixOS/nixpkgs/issues/49415#issuecomment-435611241

cdn-rsync.timer/ cdn-rsync.service, pulls in consul.service, and effectively prevents consul.service from being restarted.