AlexanderAmelkin
commented
5 years ago @geissonator, the committed fix doesn't look correct. The phosphor-ipmi-host.service unit has ExecStart, but doesn't have Type. That means that Type defaults to simple, which means that systemd will consider phosphor-ipmi-host.service successfully started right after fork'ing off, even before exec'ing the ipmid executable. So there is still a race.
Besides, we've hit a very similar intermittent problem (on a substantially older build though) and we have a very interesting observation. Pay attention to the timestamps:
This is phosphor-reset-host-check.service starting:
root@suvorov:/lib/systemd/system# systemctl status phosphor-reset-host-check@0.service --no-pager
● phosphor-reset-host-check@0.service - Check Host0 status on BMC reset
Loaded: loaded (/lib/systemd/system/obmc-host-reset@0.target.requires/../phosphor-reset-host-check@.service; disabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Fri 2019-01-18 08:55:01 UTC; 24min ago
Process: 1051 ExecStart=/usr/bin/env phosphor-host-check (code=dumped, signal=ABRT)
Main PID: 1051 (code=dumped, signal=ABRT)
Jan 18 08:54:33 suvorov systemd[1]: Starting Check Host0 status on BMC reset...
Jan 18 08:54:33 suvorov phosphor-host-check[1051]: Check if host is running
Jan 18 08:54:46 suvorov phosphor-host-check[1051]: Error in mapper call for control host
Jan 18 08:54:46 suvorov phosphor-host-check[1051]: terminate called after throwing an instance of 'std::runtime_error'
Jan 18 08:54:46 suvorov phosphor-host-check[1051]: what(): Error in mapper call for control host
Jan 18 08:55:01 suvorov systemd[1]: phosphor-reset-host-check@0.service: Main process exited, code=dumped, status=6/ABRT
Jan 18 08:55:01 suvorov systemd[1]: Failed to start Check Host0 status on BMC reset.
Jan 18 08:55:01 suvorov systemd[1]: phosphor-reset-host-check@0.service: Unit entered failed state.
Jan 18 08:55:01 suvorov systemd[1]: phosphor-reset-host-check@0.service: Failed with result 'core-dump'.And this is the mapper-wait that the prior service depends on via Wants and After:
root@suvorov:/lib/systemd/system# systemctl status mapper-wait\@-xyz-openbmc_project-control-host0.service --no-pager
● mapper-wait@-xyz-openbmc_project-control-host0.service - Wait for /xyz/openbmc_project/control/host0
Loaded: loaded (/lib/systemd/system/mapper-wait@.service; static; vendor preset: enabled)
Active: active (exited) since Fri 2019-01-18 08:55:26 UTC; 32min ago
Process: 1059 ExecStart=/usr/bin/env mapper wait /xyz/openbmc_project/control/host0 (code=exited, status=0/SUCCESS)
Main PID: 1059 (code=exited, status=0/SUCCESS)
Jan 18 08:54:35 suvorov systemd[1]: Starting Wait for /xyz/openbmc_project/control/host0...
Jan 18 08:55:26 suvorov systemd[1]: Started Wait for /xyz/openbmc_project/control/host0.As you can see, the dependant service is somehow started earlier (08:54:33) than the service it depends on (08:54:35).
This looks like a bug in systemd and doesn't look like it has anything to do with multiple services providing the same d-bus object path.
What do you think?
geissonator
@geissonator commented on Fri Jan 11 2019
I'm chasing an issue where a service intermittently fails to get a dbus object back from mapper. The service has the appropriate mapper wait in it's service file:
But still hits this intermittently:
I can see in the journal that the ipmid process (which hosts this object) had been started but it wasn't clear whether it had put the appropriate dbus object on the bus.
So I started running some tests.
I did this same test on a code image that had the old mapper (v2.4-0-gc71bcdc) and verified this test passed as expected (the mapper command hung until I started the service back up).
@geissonator commented on Fri Jan 11 2019
I'll keep looking into this until someone with more knowledge on it wants to help :)
@geissonator commented on Mon Jan 14 2019
A bit of debug points to the issue being that xyz.openbmc_project.Settings also implements /xyz/openbmc_project/control/host0
Need to see if there's a way with the mapper syntax to specify more details on what we're looking for. Otherwise may just need to encode a service dependency on phosphor-ipmi-host.service.
@geissonator commented on Mon Jan 14 2019
No obvious way to fix this within the mapper code. We discussed an optional command line parameter where you could pass the Interface but that would make the mapper service syntax very messy. For now, move this bug over to phosphor-state-manager and have the service specify explicitly which service it requires.