[wholebodydynamics] No warning or message is provided when network delays are detected.

[jeljaik]

As you now may know, we have been experiencing issues with iCubGenova01 where network delays are present rather randomly. Together with @randaz we proceeded to unplug the PC104 ethernet cable to see if this brute force event was somehow detected by WBDT but it wasn't. Given what's happening lately with Blackie, this might be quite useful and would allow torqueBalancing to put the robot in a safer state.

[traversaro]

Definitely.

A bit of detail: wholeBodyDynamicsTree uses exclusively the yarpWholeBodySensors interface to access sensor measurement, so the network delay check should be implemented there.

A good solution would be to implement a method that returns to the time since the latest measurement for each sensor, and also a method to get the oldest measurement in general. The module can then check this delay, and take the necessary action if this delay is bigger then timeout (print a warning, switch to position, etc etc).

In this way we will also get the feature ready to be used by wholeBodyEstimator.

[traversaro]

Fortunately it is much easier.. the timeout check is already included in the RemoteControlBoard implementation, so we just need to propagate the errors. Working on it right now.

If a timeout (actually, if any kind of "sensor error" is detected) the behavior is the following:

• print an error
• stop publishing estimates until the sensor reports error. This should cause the firmware to switch to HARDWARE_FAULT mode, as the firmware has a timeout check (in theory both in CAN and EMS) on torque measurement when in TORQUE control mode .

[jeljaik]

Exactly, in fact, when launching the red ball demo, for example, this is one of the consequences of a network delay, shoulders going into HARDWARE_FAULT mode. But certain modules, in particular, wholeBodyDynamics do a check on network delays, but in a way that has changed in time, and is no longer reliable (talk to randaz for more insight on this). Therefore, in the red ball demo while the arms go in hardware fault, the wholeBodyDynamics module keeps running. Then I decided to do the same check with wholeBodyDynamicsTree and the behavior was the same. Just make sure you don't do the same way wholeBodyDynamics currently checks for these delays :)

[jeljaik]

Moreover, luckily for us, while this may happen with the redBallDemo it does not happen frequently with just torqueBalancing (not that it couldn't). I guess if we launch in parallel more stuff, such as skin, or red ball demo along with it, a network delay is a possible scenario.

[traversaro]

Actually I was going to do rely on the fact that the RemoteControlBoard getEncoders method returns false if no message is received in the last 0.5 seconds (see https://github.com/robotology/yarp/blob/83466b733d89d2f3687542b047bcd5185d3f9299/src/libYARP_dev/src/modules/RemoteControlBoard/RemoteControlBoard.cpp#L74), that is exactly the method used by wholeBodyDynamics. Perhaps the timeout of 0.5 seconds is a bit to high?

For the other sensors (IMU and FT) both the WBD and WBDT ignore any timeout at the moment. Implement a timeout check on those sensor will require to modify the yarpWholeBodySensors, that is already quite a mess, but I guess is the way to go.

The alternative is to migrate both the yarpWholeBodySensors and the wholeBodyDynamics to use the AnalogSensorClient to read the FT sensors instead of directly reading from the port. The AnalogSensorClient that has a timeout check hardcoded as the RemoteControlBoard does, see https://github.com/robotology/yarp/blob/21289d209c025dbad30d02dd09047c12562e01ab/src/libYARP_dev/src/modules/AnalogSensorClient/AnalogSensorClient.h#L60, so we could rely on it to detect timeouts.

@randaz81 what changed in the wholeBodyDynamics behaviour?

[traversaro]

Fix provided in https://github.com/robotology/codyco-modules/issues/162 .

For consistency I implemented the behavior of the old wholeBodyDynamicsTree : if sensor reading fails for 10 consecutive periods, the module is automatically closed.

[traversaro]

The fix provided relies on the same checks that wholeBodyDynamics uses. I think that if we want those check to be better we should improve the yarp client devices instead of implementing high-level workarounds.

[traversaro]

TODO: migrate all of this in wholeBodyDynamics(Device/3).

[traversaro]

I am afraid this bug is still affecting wholebodydynamics YARP device.