search

openbmc / phosphor-state-manager

Apache License 2.0
11 stars 21 forks source link

synchronization target conflicts not working as expected #16

Open geissonator opened 3 years ago

geissonator commented 3 years ago

Seeing intermittent issues where the synchronization target, obmc-power-start-pre@0.target is not stopping like it should with the system is powered off.

# systemctl status obmc-power-start-pre\@0.target 
 obmc-power-start-pre@0.target - Power0 On (Pre)
     Loaded: loaded (/lib/systemd/system/obmc-power-start-pre@.target; static)
     Active: active since Fri 2021-08-13 03:06:58 UTC; 16h ago

Aug 13 03:06:58 p10bmc systemd[1]: Reached target Power0 On (Pre).

# systemctl status obmc-chassis-poweroff\@0.target 
 obmc-chassis-poweroff@0.target - Chassis0 (Power Off)
     Loaded: loaded (/lib/systemd/system/obmc-chassis-poweroff@.target; static)
     Active: active since Fri 2021-08-13 03:34:56 UTC; 16h ago

Aug 13 03:34:56 p10bmc systemd[1]: Reached target Chassis0 (Power Off).

Another synchronization target correctly stopped that conflicts with teh same target:

# systemctl status obmc-power-start\@0.target   
 obmc-power-start@0.target - Power0 On (Starting)
     Loaded: loaded (/lib/systemd/system/obmc-power-start@.target; static)
     Active: inactive (dead) since Fri 2021-08-13 03:34:48 UTC; 16h ago

Aug 13 03:23:06 p10bmc systemd[1]: Reached target Power0 On (Starting).
Aug 13 03:34:48 p10bmc systemd[1]: Stopped target Power0 On (Starting).
geissonator commented 3 years ago

The only major difference between the working and non-working target is this is in the non-working:

Wants=mapper-subtree-remove@-xyz-openbmc\x5fproject-software\x3Axyz.openbmc_project.Software.ActivationBlocksTransition.service
After=mapper-subtree-remove@-xyz-openbmc\x5fproject-software\x3Axyz.openbmc_project.Software.ActivationBlocksTransition.service
Wants=mapper-subtree-remove@-xyz-openbmc\x5fproject-logging\x3Axyz.openbmc_project.Logging.ErrorBlocksTransition.service
After=mapper-subtree-remove@-xyz-openbmc\x5fproject-logging\x3Axyz.openbmc_project.Logging.ErrorBlocksTransition.service

My hunch at this point is that our somewhat unorthodox use of Wants/After using mapper is causing a bug of some sort in systemd. Looking at the journal, I see those services starting and running as expected, even in the error scenarios, so nothing obvious yet.

geissonator commented 3 years ago

Workaround put in https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-power/+/45893, remove once this issue figured out.