search

ami-iit / yarp-openmct

Repo for YARP and OpenMCT integration.
BSD 3-Clause "New" or "Revised" License
6 stars 1 forks source link

Telemetry server crashes at startup when parsing `<robot>/left_foot_heel_tiptoe/measures:o` port FT data #146

Closed nunoguedelha closed 1 year ago

nunoguedelha commented 1 year ago

Refer to [Issue 2] in issue recap below:

Test Report

Objectives

The tests focused on checking the data pipe and overall functionality related to the Yarp Text Logging of the yarprobotinterface process running on the robot head and the module WalkingModule running on some other machine connected to the robot network. The related telemetry domain entries in the Open MCT Visualiser, yarprobotinterface process logging through yarplogger and WalkingModule process logging through yarplogger under the telemetry folder Process Logging: image

Test Sequence, Outcome and Issues

  • Installation went smoothly.

  • [Issue 1]: Running the telemetry server from the folder ./iCubTelemVizServer and node command fails:

    $> cd iCubTelemVizServer
$> node iCubTelemVizServer.js
...
/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282
           this.state[id].ori.roll = sensorSample[3][0][0][0];
                                                       ^

TypeError: Cannot read property '0' of undefined
   at ICubTelemetry.parseFromId (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282:57)
   at Object.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:166:31)
   at /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js:74:56
   at /Users/nunoguedelha/dev/yarp-openmct/node_modules/YarpJS/yarp.js:188:13
$>
    Full log ``` iCub Telemetry server launched! [INFO] |yarp.os.Port|/yarpjs/inertial:i| Port /yarpjs/inertial:i active at tcp://192.168.1.70:10130/ [INFO] |yarp.os.Port|/yarpjs/head/inertials:i| Port /yarpjs/head/inertials:i active at tcp://192.168.1.70:10131/ [INFO] |yarp.os.Port|/yarpjs/leftArm/inertials:i| Port /yarpjs/leftArm/inertials:i active at tcp://192.168.1.70:10132/ [INFO] |yarp.os.Port|/yarpjs/rightArm/inertials:i| Port /yarpjs/rightArm/inertials:i active at tcp://192.168.1.70:10133/ [INFO] |yarp.os.Port|/yarpjs/leftLeg/inertials:i| Port /yarpjs/leftLeg/inertials:i active at tcp://192.168.1.70:10134/ [INFO] |yarp.os.Port|/yarpjs/rightLeg/inertials:i| Port /yarpjs/rightLeg/inertials:i active at tcp://192.168.1.70:10135/ [INFO] |yarp.os.Port|/yarpjs/leftFoot/inertials:i| Port /yarpjs/leftFoot/inertials:i active at tcp://192.168.1.70:10136/ [INFO] |yarp.os.Port|/yarpjs/rightFoot/inertials:i| Port /yarpjs/rightFoot/inertials:i active at tcp://192.168.1.70:10137/ [INFO] |yarp.os.Port|/yarpjs/left_leg/stateExt:o| Port /yarpjs/left_leg/stateExt:o active at tcp://192.168.1.70:10138/ [INFO] |yarp.os.Port|/yarpjs/right_leg/stateExt:o| Port /yarpjs/right_leg/stateExt:o active at tcp://192.168.1.70:10139/ [INFO] |yarp.os.Port|/yarpjs/left_arm/stateExt:o| Port /yarpjs/left_arm/stateExt:o active at tcp://192.168.1.70:10140/ [INFO] |yarp.os.Port|/yarpjs/right_arm/stateExt:o| Port /yarpjs/right_arm/stateExt:o active at tcp://192.168.1.70:10141/ [INFO] |yarp.os.Port|/yarpjs/torso/stateExt:o| Port /yarpjs/torso/stateExt:o active at tcp://192.168.1.70:10142/ [INFO] |yarp.os.Port|/yarpjs/head/stateExt:o| Port /yarpjs/head/stateExt:o active at tcp://192.168.1.70:10143/ [INFO] |yarp.os.Port|/yarpjs/camLeftEye:i| Port /yarpjs/camLeftEye:i active at tcp://192.168.1.70:10144/ [INFO] |yarp.os.Port|/yarpjs/camRightEye:i| Port /yarpjs/camRightEye:i active at tcp://192.168.1.70:10145/ [INFO] |yarp.os.Port|/yarpjs/left_arm/cartesianEndEffectorWrench:i| Port /yarpjs/left_arm/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10146/ [INFO] |yarp.os.Port|/yarpjs/right_arm/cartesianEndEffectorWrench:i| Port /yarpjs/right_arm/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10147/ [INFO] |yarp.os.Port|/yarpjs/left_upper_leg/cartesianEndEffectorWrench:i| Port /yarpjs/left_upper_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10148/ [INFO] |yarp.os.Port|/yarpjs/left_lower_leg/cartesianEndEffectorWrench:i| Port /yarpjs/left_lower_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10149/ [INFO] |yarp.os.Port|/yarpjs/right_upper_leg/cartesianEndEffectorWrench:i| Port /yarpjs/right_upper_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10150/ [INFO] |yarp.os.Port|/yarpjs/right_lower_leg/cartesianEndEffectorWrench:i| Port /yarpjs/right_lower_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10151/ [INFO] |yarp.os.Port|/yarpjs/left_foot_front/cartesianEndEffectorWrench:i| Port /yarpjs/left_foot_front/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10152/ [INFO] |yarp.os.Port|/yarpjs/left_foot_rear/cartesianEndEffectorWrench:i| Port /yarpjs/left_foot_rear/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10153/ [INFO] |yarp.os.Port|/yarpjs/right_foot_front/cartesianEndEffectorWrench:i| Port /yarpjs/right_foot_front/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10154/ [INFO] |yarp.os.Port|/yarpjs/right_foot_rear/cartesianEndEffectorWrench:i| Port /yarpjs/right_foot_rear/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10155/ [INFO] |yarp.os.Port|/yarpjs/left_arm/FT:i| Port /yarpjs/left_arm/FT:i active at tcp://192.168.1.70:10156/ [INFO] |yarp.os.Port|/yarpjs/right_arm/FT:i| Port /yarpjs/right_arm/FT:i active at tcp://192.168.1.70:10157/ [INFO] |yarp.os.Port|/yarpjs/left_leg_hip/FT:i| Port /yarpjs/left_leg_hip/FT:i active at tcp://192.168.1.70:10158/ [INFO] |yarp.os.Port|/yarpjs/right_leg_hip/FT:i| Port /yarpjs/right_leg_hip/FT:i active at tcp://192.168.1.70:10159/ [INFO] |yarp.os.Port|/yarpjs/left_foot_heel_tiptoe/FT:i| Port /yarpjs/left_foot_heel_tiptoe/FT:i active at tcp://192.168.1.70:10160/ [INFO] |yarp.os.Port|/yarpjs/right_foot_heel_tiptoe/FT:i| Port /yarpjs/right_foot_heel_tiptoe/FT:i active at tcp://192.168.1.70:10161/ [INFO] |yarp.os.Port|/yarpjs/battery/data:i| Port /yarpjs/battery/data:i active at tcp://192.168.1.70:10162/ [INFO] |yarp.os.Port|/yarpjs/walking-coordinator-logger/data:i| Port /yarpjs/walking-coordinator-logger/data:i active at tcp://192.168.1.70:10163/ [INFO] |yarp.os.Port|/yarpjs/proc-yarprobotinterface-yarptextlogging:i| Port /yarpjs/proc-yarprobotinterface-yarptextlogging:i active at tcp://192.168.1.70:10164/ [INFO] |yarp.os.Port|/yarpjs/proc-walkingmodule-yarptextlogging:i| Port /yarpjs/proc-walkingmodule-yarptextlogging:i active at tcp://192.168.1.70:10165/ [INFO] |yarp.os.Port|/yarpjs/sysCmdsGenerator/rpc| Port /yarpjs/sysCmdsGenerator/rpc active at tcp://192.168.1.70:10166/ { status: 'OK', message: 'Opem-MCT static server process started.' } info:Yarp command successfully started... ICubTelemetry History Server listening on http://192.168.1.70:8081/history ICubTelemetry Realtime Server listening on ws://192.168.1.70:8081/realtime Control Console Server listening on http://192.168.1.70:3000 Yarp port names retrieval completed successfully (0) [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/inertial:i| Receiving input from /icubSim/inertial to /yarpjs/inertial:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/head/inertials:i| Receiving input from /icubSim/head/inertials/measures:o to /yarpjs/head/inertials:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_leg/stateExt:o| Receiving input from /icubSim/left_leg/stateExt:o to /yarpjs/left_leg/stateExt:o using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_leg/stateExt:o| Receiving input from /icubSim/right_leg/stateExt:o to /yarpjs/right_leg/stateExt:o using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_arm/stateExt:o| Receiving input from /icubSim/left_arm/stateExt:o to /yarpjs/left_arm/stateExt:o using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_arm/stateExt:o| Receiving input from /icubSim/right_arm/stateExt:o to /yarpjs/right_arm/stateExt:o using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/torso/stateExt:o| Receiving input from /icubSim/torso/stateExt:o to /yarpjs/torso/stateExt:o using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/head/stateExt:o| Receiving input from /icubSim/head/stateExt:o to /yarpjs/head/stateExt:o using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_arm/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_arm/cartesianEndEffectorWrench:o to /yarpjs/left_arm/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_arm/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_arm/cartesianEndEffectorWrench:o to /yarpjs/right_arm/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_upper_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_upper_leg/cartesianEndEffectorWrench:o to /yarpjs/left_upper_leg/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_lower_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_lower_leg/cartesianEndEffectorWrench:o to /yarpjs/left_lower_leg/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_upper_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_upper_leg/cartesianEndEffectorWrench:o to /yarpjs/right_upper_leg/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_lower_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_lower_leg/cartesianEndEffectorWrench:o to /yarpjs/right_lower_leg/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_foot_front/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_foot_front/cartesianEndEffectorWrench:o to /yarpjs/left_foot_front/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_foot_rear/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_foot_rear/cartesianEndEffectorWrench:o to /yarpjs/left_foot_rear/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_foot_front/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_foot_front/cartesianEndEffectorWrench:o to /yarpjs/right_foot_front/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_foot_rear/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_foot_rear/cartesianEndEffectorWrench:o to /yarpjs/right_foot_rear/cartesianEndEffectorWrench:i using tcp [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/proc-yarprobotinterface-yarptextlogging:i| Receiving input from /log/iiticublap199/yarprobotinterfa[yarprobotinterface]/85559 to /yarpjs/proc-yarprobotinterface-yarptextlogging:i using tcp [OPEN-MCT STATIC SERVER] ipc: {"pid":85881} [INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/proc-walkingmodule-yarptextlogging:i| Receiving input from /log/iiticublap199/WalkingModule-0.[walkingModule]/85560 to /yarpjs/proc- walkingmodule-yarptextlogging:i using tcp /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282 this.state[id].ori.roll = sensorSample[3][0][0][0]; ^ TypeError: Cannot read property '0' of undefined at ICubTelemetry.parseFromId (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282:57) at Object. (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:166:31) at /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js:74:56 at /Users/nunoguedelha/dev/yarp-openmct/node_modules/YarpJS/yarp.js:188:13 ```
  • The app could be launched without issues from the repo root: 
    npm start

  • [Issue 2]: As soon as the server starts, it crashes with the following issue:

    /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404
   this.history[id].push(telemetrySample); // update history
                    ^

TypeError: Cannot read property 'push' of undefined
   at ICubTelemetry.generateTelemetry (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404:22)
   at ICubTelemetry.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:183:18)
   at Array.forEach (<anonymous>)
   at ICubTelemetry.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:182:33)
   at listOnTimeout (internal/timers.js:555:17)
   at processTimers (internal/timers.js:498:7)
    Extended log ``` NEW CONNECTION!! CLOSE CONNECTION [OPEN-MCT STATIC SERVER] stdout: CLOSE CONNECTION NEW CONNECTION!! CLOSE CONNECTION [INFO] |yarp.os.impl.PortCoreInputUnit| Removing input from /log/iiticublap199/WalkingModule-0.[walkingModule]/85560 to /yarpjs/proc-walkingmodule-yarptextlogging:i [INFO] |yarp.os.impl.PortCoreInputUnit| Receiving input from /icubSim/left_foot_heel_tiptoe/measures:o to /yarpjs/left_foot_heel_tiptoe/FT:i using tcp /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404 this.history[id].push(telemetrySample); // update history ^ TypeError: Cannot read property 'push' of undefined at ICubTelemetry.generateTelemetry (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404:22) at ICubTelemetry. (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:183:18) at Array.forEach () at ICubTelemetry. (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:182:33) at listOnTimeout (internal/timers.js:555:17) at processTimers (internal/timers.js:498:7) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! yarp-openmct@1.0.0 start: `cd iCubTelemVizServer && node ${NODE_DEBUG_OPTION} iCubTelemVizServer.js` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the yarp-openmct@1.0.0 start script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! /Users/nunoguedelha/.npm/_logs/2022-10-04T08_39_39_709Z-debug.log ```
  • The issue was fixed with a debug branch debug/test-with-stefano pushed to the remote repository. Meanwhile, the robot had to go to maintenance.
  • [Issue 3]: Later in the day, @S-Dafarra ran another trial with the debug branch. The issue observed earlier was fixed, no crash ever occurred. We could see the expected Yarplogger messages on the visualiser, but there was a glitch though:
    • As we ran some rpc command (e.g. calib all 300) triggering Yarplogger messages we could not see the messages right away.
    • Later on, we could only see the messages if we clicked on another entry in the left pane, then on the Yarp Text Logging entry back again.
    • There is no log for this issue.

Issues recap

  • [ ] [Issue 1]: Running the telemetry server from the folder ./iCubTelemVizServer with the node command fails
  • [ ] [Issue 2]: Telemetry server crashes at startup when parsing <robot>/left_foot_heel_tiptoe/measures:o port FT data.
  • [ ] [Issue 3]: We can't see the Yarplogger messages right away as they are received on the yarpjs input port, unless we click away from the Yarp Text Logging entry and back again.

_Originally posted by @nunoguedelha in https://github.com/ami-iit/component_ANA-Avatar-XPRIZE/issues/760#issuecomment-1266616475_

Fixed in #148 .

nunoguedelha commented 1 year ago

The issue is due to a mismatch of the telemetry ID related to the FT feet measurements, between the value used in the state variable and the value associated to the YARP port and received samples

Here's the telemetry ID in the state variable... https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L117-L120

...versus the one associated to the YARP port and received samples https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/iCubTelemVizServer.js#L68-L76

The issue was introduced by https://github.com/ami-iit/yarp-openmct/pull/136 and related further fixes.

Background

The FTs attached to the feet heels and toes have their data published on the same ports: left / right heel and toe FTs data published respectively on /<robotName>/left_foot_heel_tiptoe/measures:o / /<robotName>/right_foot_heel_tiptoe/measures:o.

For having a common format for all the FTs transmitted data and dictionary metadata, we parse the YARP data and send the each FT chunk of data separately. For that purpose, we define a single telemetry ID for identifying each port, namely sens.leftFootHeelTiptoeFTs and sens.rightFootHeelTiptoeFTs... https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/conf/servers.json#L318-L320 https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/conf/servers.json#L328-L330

...same IDs used here... https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/iCubTelemVizServer.js#L71-L76

...and here... https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L245 https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L351 https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L356

...but separate IDs for the telemetry samples stored in icubtelemetry.state, icubtelemetry.history: https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L351-L360 https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L246-L254

The Issue

Separate IDs should also be used for sending the data to the client but instead we use the port related key: https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L161-L164

https://github.com/ami-iit/yarp-openmct/blob/b010f317433f66fc7775e1188b473f239580fc8c/iCubTelemVizServer/icubtelemetry.js#L147-L152

the reported error occurs in the code block above.

nunoguedelha commented 1 year ago

Fix

nunoguedelha commented 1 year ago

Fixed in #148 .