search

openbmc / phosphor-pid-control

OpenBMC PID-based Thermal Control Daemon
Apache License 2.0
16 stars 21 forks source link

Sometimes return "Connection timed out" when set auto mode continuously #30

Open harveyquta opened 1 year ago

harveyquta commented 1 year ago

When set auto mode continuously, it will sometimes return "Connection timed out". But when set to manual mode, it won't happen.

set auto mode cmd: busctl set-property xyz.openbmc_project.State.FanCtrl /xyz/openbmc_project/settings/fanctrl/zone2 xyz.openbmc_project.Control.Mode Manual b false; \ busctl set-property xyz.openbmc_project.State.FanCtrl /xyz/openbmc_project/settings/fanctrl/zone0 xyz.openbmc_project.Control.Mode Manual b false; \ busctl set-property xyz.openbmc_project.State.FanCtrl /xyz/openbmc_project/settings/fanctrl/zone1 xyz.openbmc_project.Control.Mode Manual b false;

return: Failed to set property Manual on interface xyz.openbmc_project.Control.Mode: Connection timed out

But if adding sleep between each busctl cmd, like below, the error will not happen. busctl set-property xyz.openbmc_project.State.FanCtrl /xyz/openbmc_project/settings/fanctrl/zone2 xyz.openbmc_project.Control.Mode Manual b false; sleep 0.1; \ busctl set-property xyz.openbmc_project.State.FanCtrl /xyz/openbmc_project/settings/fanctrl/zone0 xyz.openbmc_project.Control.Mode Manual b false; sleep 0.1; \ busctl set-property xyz.openbmc_project.State.FanCtrl /xyz/openbmc_project/settings/fanctrl/zone1 xyz.openbmc_project.Control.Mode Manual b false;

harveyquta commented 1 year ago

if set manual mode continuously, the error also will not happen.

Krellan commented 1 year ago

Very strange. I wonder what would be causing this?

williamspatrick commented 1 year ago

Daemon blocked on sysfs calls to update fan speeds and not able to respond to dbus fast enough?

harveyquta commented 1 year ago

Seems like what @williamspatrick said. After I mark the dbus_call in dbus/dbuswrite.cpp, "Connection timed out" will not happen. When set manual to false, _redundantWrite will be set to true, and pid will set property mandatorily one time.

Krellan commented 1 year ago

This is a general problem, not just with the setting of automatic mode.

Sometimes D-Bus calls are just straight up forgotten about. The caller receives the timeout. The swampd program receives nothing at all, so it takes no action.

This problem happens rarely enough that it is tricky to debug. Any thoughts?