canopen_master "TimeoutException" with Schneider LXM28A (CiA 402) and RevPi Connect

[riv-robot]

I've got a blocker on canopen_master (CANopen communication). The current state is that all plumbing is done, both controller and CAN bus seems to be ok but the ros_canopen fails on launch. I have just one device, a Schneider LXM28A and my CAN master is a RevPi Connect with ConCAN module. Attached are the following:

• stderr.log (ros_canopen node)

  [ERROR] [1619040861.136742926]: EMCY received: 81#10ffa10000001500
  [ERROR] [1619040861.144943677]: EMCY received: 81#9170a10000001300
  [ERROR] [1619040862.143727648]: Did not receive a response message
  [ERROR] [1619040862.145883505]: Received error frame: 20000004#0008000000006501
  [ERROR] [1619040862.146420374]: Received error frame: 20000004#0020000000008501
  [ERROR] [1619040862.148657136]: Initializing failed: /home/pi/ros_catkin_ws/src/ros_canopen/canopen_master/src/sdo.cpp(429): Throw in function void canopen::SDOClient::transmitAndWait(const canopen::ObjectDict::Entry&, const canopen::String&, canopen::String*)
  Dynamic exception type: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<canopen::TimeoutException> >
  std::exception::what: SDO
  [canopen::tag_objectdict_key*] = 1800sub1

• stdout.log (ros_canopen node)

  [ INFO] [1619040844.788885087]: Using fixed control period: 0.010000000
  [ INFO] [1619040861.113530355]: Current state: 1 device error: system:0 internal_error: 0 (OK)
  [ INFO] [1619040861.113997556]: Current state: 2 device error: system:0 internal_error: 0 (OK)
  [ INFO] [1619040862.146848900]: Current state: 2 device error: system:125 internal_error: 0 (OK)
  [ INFO] [1619040862.147099477]: Current state: 0 device error: system:125 internal_error: 0 (OK)
  [ INFO] [1619040862.147391027]: Current state: 0 device error: system:0 internal_error: 0 (OK)
  [ INFO] [1619040862.147571424]: Current state: 0 device error: system:0 internal_error: 0 (OK)

• candump (with timestamps)

  (1619074296.791695)  can0  000   [2]  82 01
  (1619074296.813944)  can0  701   [1]  00
  (1619074296.814044)  can0  081   [8]  10 FF A1 00 00 00 15 00
  (1619074296.814587)  can0  601   [8]  2B 17 10 00 00 00 00 00
  (1619074296.821427)  can0  581   [8]  60 17 10 00 00 00 00 00
  (1619074296.822007)  can0  081   [8]  91 70 A1 00 00 00 13 00
  (1619074296.822274)  can0  601   [8]  01 00 00 01 00 00 00 00
  (1619074297.822919)  can0  601   [8]  00 00 00 01 D8 66 E5 01
  (1619074297.824384)  can0  000   [2]  02 01

• sdo.cpp (canopen_master)

  • this is a code snippet of where it fails

    
    void SDOClient::transmitAndWait(const canopen::ObjectDict::Entry &entry, const String &data,  String *result){
    buffer = data;
    offset = 0;
    total = buffer.size();
    current_entry = &entry;
    done = false;

  can::BufferedReader::ScopedEnabler enabler(reader_);

  if(result){ interface_->send(last_msg = UploadInitiateRequest(clientid, entry)); }else{ interface->send(last_msg = DownloadInitiateRequest(client_id, entry, buffer, offset)); }

  boost::this_thread::disable_interruption di; can::Frame msg;

  while(!done){ if(!reader_.read(&msg,boost::chrono::seconds(1))) { abort(0x05040000); // SDO protocol timed out. ROSCANOPEN_ERROR("canopen_master", "Did not receive a response message"); break; } if(!processFrame(msg)){ ROSCANOPEN_ERROR("canopen_master", "Could not process message"); break; } } if(offset == 0 || offset != total){ THROW_WITH_KEY(TimeoutException("SDO"), ObjectDict::Key(*current_entry)); }

  if(result) *result=buffer;

}

- **[positioner_driver.yaml](https://github.com/additiveautomations/aa-canopen/blob/master/positioner_driver.yaml)** & **[EDS](https://github.com/additiveautomations/aa-canopen)** (config files)
    - Note I have mapped the drive operation modes (ID 1 only) and the other mappings that were recommended [here](http://wiki.ros.org/canopen_402?distro=noetic)
- Vector **CANeds** EDS check  
![Vector CANeds check](https://user-images.githubusercontent.com/44445100/115671857-da442480-a342-11eb-9160-ec7f3685b299.png)

[gavanderhoorn]

The link was more meant as an example of a potential issue, not specific to this case.

[mathias-luedtke]

Nothing else is running on the CAN bus other than the RevPi ConCAN module which acts as my hardware interface.

@robertjbush: And no other process that accesses the can0 (or whatever it is called in your system) interface?

[riv-robot]

@robertjbush: And no other process that accesses the can0 (or whatever it is called in your system) interface?

Nope, I ran find /proc/* -type f | grep can0$ | head and netstat -ap -i can0, sudo ss -tulpn. Output from sudo netstat -tulpn.

[mathias-luedtke]

Nope, I ran find /proc/* -type f | grep can0$ | head and netstat -ap -i can0, sudo ss -tulpn. Output from sudo netstat -tulpn.

Does any of them show the canopen_motor_node process?

[riv-robot]

@ipa-mdl See below for output (it shows that the canopen_motor_node process does indeed access the interface).
I am still stuck with the "halt" error....

[mathias-luedtke]

See below for output (it shows that the canopen_motor_node process does indeed access the interface).

This only lists the TCP ports, which are used by ROS, but not CAN. Perhaps we have to log all CAN messages processed and sent by canopen_motor_node..

I am still stuck with the "halt" error....

Have you tried another switching state?

[riv-robot]

Success! 🟢🟢🟢
It has initialised successfully, finally...
I decided to go back to one of the 30 eds files I had tried (one I had labelled _BEST.dcf from last weekend) and go back to a bare bones configuration (i.e. no heartbeat, not specifying any 'advanced' flags, even no switching state).

⛔ Still not moving because moveit! can't recognise my ros controller but that's off topic.

I'd like to get to the bottom of this issue as I feel it's never good to leave errors in the background. If there's anything you need @ipa-mdl to be able to debug it further, just write it down here and I'll do it for you. The team doesn't have the expertise with C++ to debug further (not easy to find expert C++ engineers!) so I can't do much more until we hire someone.

[mathias-luedtke]

Great to hear!

If there's anything you need @ipa-mdl to be able to debug it further, just write it down here and I'll do it for you.

I might have some time this week end to design a test case for the SDO communication on ARM. Including a trace option. I will keep you posted!

[gavanderhoorn]

So what's the conclusion here (for now)?

@robertjbush: what was it that made it suddenly work?

[riv-robot]

@gavanderhoorn

Conclusion

• Failure Mode:
  • timeout exception when writing/reading SDO's in objection dictionary key 1800sub1
• Effect:
  • cannot re-map PDO's even though configuration and .dcf are correct
• Detection:
  • place debug statements in sdo.cpp, line 305, 308, 310 and 429. If offset, buffer and total are not constant then it's the same issue.
• Root Cause:
  • TBC, but most likely to be source code of canopen_master rather than external factors
• Corrective Action:
  • in sdo.cpp comment out line 429
• Corrective Improvement:
  • TBC, once root cause determined

[mathias-luedtke]

Root Cause: TBC, but most likely to be source code of canopen_master rather than external factors

I have added a test case (#428) and it passed on ARM64 as well. Please test this branch locally as well (catkin_make run_tests), without commenting out anything..

Corrective Action: in sdo.cpp comment out line 429

Commenting out the error detection is not a solution.

[riv-robot]

Commenting out the error detection is not a solution.

You can read Read FMEA definitions here (see books). Corrective Action is not a solution by definition.

Please test this branch locally as well (catkin_make run_tests), without commenting out anything..

Will do as soon as I get my motor moving.

[mathias-luedtke]

You can read Read FMEA definitions here (see books). Corrective Action is not a solution by definition.

Let me rephrase then: It is not a proper corrective action either. Instead mitigating the failure (or its effect), you just comment the singe line which would report it to the caller. This is like painting the error light black.

Will do as soon as I get my motor moving.

I truly understand that this is your priority, but if SDOs are not working properly this might affect the motion signals. Especially since we cannot rule out (yet) that there is some issue with memory alignment, which would be present for PDOs as well.

[riv-robot]

Ok understood, so:

1. Installed the branch from #428.
2. Error persists on normal operation.
3. catkin_make run_tests for ros_canopen no errors.
4. catkin_make_isolated --catkin-make-args run_tests does give a worrying error and failed before completion (in tf2):
   

My plan of action is:

1. Wipe RevPi completely
2. Install the very first release (last week) of Buster for the RevPi
3. Reinstall everything

4. Try again

   As a general note, I wish I had these tests earlier or that I would have thought about doing them earlier because they help in debugging system installation and package installation. A note on the ros_canopen wiki page would help those with less experience, like myself.

[riv-robot]

💚 Updated the RevPi image to Buster and the issue disappeared. For those who own a RevPi (Connect, Core, Compact, Flat) and want to use ROS make sure you take advantage of last weeks much overdue release of Buster.

The test suite was very helpful in debugging, thank you @ipa-mdl, please add it to the package and wiki for future users. And likewise the catkin workspace advice @gavanderhoorn was very helpful in setting up my fresh install. Its way quicker and robust!

To finish on a negative note, publishing to the command topic /canopen/positioner_controller/command still just hangs and doesn't execute a position based move....I suppose its time to open a ROS answers thread as this one can be closed.