Segmentation fault in Torque Control

[yepw]

Description

j2s7s300_driver crashes in torque control. It happens only with Ethernet connection. I tried to use gdb to backtrace the error:

[ INFO] [1654995649.967037899]: Switching to torque control
[ WARN] [1654995657.833677076]: destruction entered!
[Thread 0x7fffd67fc700 (LWP 8060) exited]
[Thread 0x7fffd6ffd700 (LWP 8059) exited]
[Thread 0x7fffd77fe700 (LWP 8058) exited]
[Thread 0x7fffd7fff700 (LWP 8057) exited]
[Thread 0x7fffeca1b700 (LWP 8055) exited]
[Thread 0x7fffeda1d700 (LWP 8053) exited]

Thread 1 "kinova_arm_driv" received signal SIGSEGV, Segmentation fault.
_dl_close (_map=0x0) at dl-close.c:819
819 dl-close.c: No such file or directory.
(gdb) backtrace
#0  _dl_close (_map=0x0) at dl-close.c:819
#1  0x00007ffff63e416f in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffa580, 
    operate=0x7ffff3f2a070 <dlclose_doit>, args=0x0) at dl-error-skeleton.c:196
#2  0x00007ffff63e41ff in __GI__dl_catch_error (objname=0x5555557903b0, errstring=0x5555557903b8, 
    mallocedp=0x5555557903a8, operate=<optimized out>, args=<optimized out>) at dl-error-skeleton.c:215
#3  0x00007ffff3f2a745 in _dlerror_run (operate=operate@entry=0x7ffff3f2a070 <dlclose_doit>, args=0x0) at dlerror.c:162
#4  0x00007ffff3f2a0b3 in __dlclose (handle=<optimized out>) at dlclose.c:46
#5  0x00007fffedc464b4 in Ethernet_CloseAPI () at ../Kinova.API.EthCommandLayerUbuntu.cpp:698
#6  0x00007ffff6ea4fdd in kinova::KinovaComm::~KinovaComm() () from /home/uwgraphics/jaco_ws/devel/lib/libkinova_driver.so
#7  0x000055555556375d in main ()
(gdb) 

Version

ROS distribution : melodic Branch and commit you are using : melodic-devel

Steps to reproduce

1. Connect to a jaco2 7DoF arm with Ethernet
2. roslaunch kinova_bringup kinova_robot.launch kinova_robotType:=j2s7s300
3. rosservice call /j2s7s300_driver/in/set_torque_control_mode "state: 1"
4. rostopic pub -r 100 /j2s7s300_driver/in/joint_torque kinova_msgs/JointTorque "{joint1: 0.0, joint2: 0.0, joint3: 0.0, joint4: 0.0, joint5: 0.0, joint6: 0.0, joint7: 0.0}
5. j2s7s300_driver node crashed. [j2s7s300_driver-2] process has died [pid 6291, exit code -11, cmd /home/uwgraphics/jaco_ws/devel/lib/kinova_driver/kinova_arm_driver j2s7s300 __name:=j2s7s300_driver __log:=/home/uwgraphics/.ros/log/2b49b4ba-e9e9-11ec-b323-e4a471f97180/j2s7s300_driver-2.log].

Expected behavior

I wanted the robot to enter "zero gravity" mode and to move the robot by manually dragging it.

Any other information

The same error occurred when running rosrun kinova_demo gravity_compensated_mode.py j2s7s300. However, when running gravity_compensated_mode.py, the robot did successfully enter the gravity compensated mode before the driver crashed.

My robot_parameters.yaml

# Set this parameter to use a specific arm on your system    --> 
# serial_number: PJ00000001030703130

# Joint speed limit for joints 1, 2, 3
jointSpeedLimitParameter1: 10

# Joint speed limit for joints 4, 5, 6
jointSpeedLimitParameter2: 20

# payload: [COM COMx COMy COMz]
payload: [1.0, 0, 0, 0.05]

connection_type: Ethernet #USB

# Ethernet connection parameters
ethernet: {
  local_machine_IP: 192.168.100.100,
  subnet_mask: 255.255.255.0,
  local_cmd_port: 25015,
  local_broadcast_port: 25025
}

#Torque control parameters
#Do not change these parameters unless you want to change torque control behavior
torque_parameters: {
      publish_torque_with_gravity_compensation: true,
      use_estimated_COM_parameters: false,
    # if torque min/max sepecified, all min/max values need to be specified
    #  torque_min: [80, 80, 80, 80, 80, 80, 80],
    #  torque_max: [90, 90, 90, 90, 90, 90, 90],
    # Decides velocity threshold at which robot switches torque to position control (between 0 and 1)
    safety_factor: 1,
    # COM parameters
    # order [m1,m2,...,m7,x1,x2,...,x7,y1,y2,...y7,z1,z2,...z7]
    # com_parameters: [0,0,0,0,0,0,0, 0,0,0,0,0,0,0, 0,0,0,0,0,0,0, 0,0,0,0,0,0,0]
}

[yepw]

To provide more information, the firmware version of our Jaco arm is 6.2.10.0.

Also, I got the same error in ROS noetic with the noetic-devel branch.

Please let me know if I could provide more information to help you fix the issue.

[felixmaisonneuve]

Hi @yepw,

After looking into it, it seems to be an issue only with the Ethernet connection. Did you remove the gripper on your arm? There is a verification in the firmware's code that validates the message length (data must contain torque value for 6 or 7 joints + unused data for each finger). This verification is not present for the USB commands. This might explain why it works on USB but not Ethernet.

I did not manage to test this yet, and we are currently looking into this issue. I will keep you updated when more progress is made.

In the mean time, feel free to post any interesting test or behaviour you might experience.

Best, Felix

[yepw]

Thanks, @felixmaisonneuve The error might be relevant to the gripper because I am using a Robotiq 1DoF gripper. I wonder how to pass the verification. I tried rostopic pub -r 100 /j2n6s300_driver/in/joint_torque kinova_msgs/JointTorque "{joint1: 0.0, joint2: 0.0, joint3: 0.0, joint4: 0.0, joint5: 0.0, joint6: 0.0, joint7: 0.0, finger1: 0.0, finger2: 0.0, finger3: 0.0}". This didn't work because the kinova_msgs/JointTorque message type only contains joint torques. The error I got:

ERROR: No field name [finger3]

Args are: [joint1 joint2 joint3 joint4 joint5 joint6 joint7]

[felixmaisonneuve]

There is no way for you to bypass this, the probleme is in the base firmware (If the issue is what I think it is). If you want, I encourage you to contact support@kinova.ca and we can send you the test firmware so you can try it on your setup. If you do, please mention you want to test firmware and mention my name so whoever get your email will be able link it to me.

That is probably the fastest way for you to get results.

Best, Felix

[yepw]

Thanks, @felixmaisonneuve. I sent an email to support@kinova.ca a few days ago and haven't heard back. I did mention your name and test firmware in the email. My email address is yeping@cs.wisc.edu No sure whether the email was linked to you or not. Please let me know if you need me to send another email.

[yepw]

@felixmaisonneuve Thank you. The custom build firmware you sent to me addresses the problem!