Cannot run the examples launch file

oruiz97 commented 3 years ago

I've been taking a look at this package and wanted to launch the example to start experimenting on how to use it. However I have not been able to run it. These have been my steps:

Cloning the package into a new catkinws/src folder
Building as stated in the readme catkin_make -DCMAKE_BUILD_TYPE=Release
source devel/setup.bash
roslaunch cartesian_controller_examples examples.launch

I am using ROS noetic in a debian machine. This is the error message that I get. Is there a step I am missing? I don't see any further indication of that in the readme.

Error message after launching examples.launch

```log ... logging to /home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/roslaunch-acuari-6157.log Checking log directory for disk usage. This may take a while. Press Ctrl-C to interrupt Done checking log file disk usage. Usage is <1GB. xacro: in-order processing became default in ROS Melodic. You can drop the option. started roslaunch server http://acuari:35239/ SUMMARY ======== PARAMETERS * /generic_hw_control_loop/cycle_time_error_threshold: 0.03 * /generic_hw_control_loop/loop_hz: 100 * /hardware_interface/joints: ['joint1', 'joint... * /hardware_interface/sim_control_mode: 0 * /joint_state_controller/publish_rate: 50 * /joint_state_controller/type: joint_state_contr... * /joint_trajectory_controller/joints: ['joint1', 'joint... * /joint_trajectory_controller/type: position_controll... * /my_cartesian_compliance_controller/compliance_ref_link: tool0 * /my_cartesian_compliance_controller/end_effector_link: tool0 * /my_cartesian_compliance_controller/ft_sensor_ref_link: sensor_link * /my_cartesian_compliance_controller/joints: ['joint1', 'joint... * /my_cartesian_compliance_controller/pd_gains/rot_x/p: 0.01 * /my_cartesian_compliance_controller/pd_gains/rot_y/p: 0.01 * /my_cartesian_compliance_controller/pd_gains/rot_z/p: 0.01 * /my_cartesian_compliance_controller/pd_gains/trans_x/p: 0.05 * /my_cartesian_compliance_controller/pd_gains/trans_y/p: 0.05 * /my_cartesian_compliance_controller/pd_gains/trans_z/p: 0.05 * /my_cartesian_compliance_controller/robot_base_link: base_link * /my_cartesian_compliance_controller/stiffness/rot_x: 100 * /my_cartesian_compliance_controller/stiffness/rot_y: 100 * /my_cartesian_compliance_controller/stiffness/rot_z: 100 * /my_cartesian_compliance_controller/stiffness/trans_x: 500 * /my_cartesian_compliance_controller/stiffness/trans_y: 500 * /my_cartesian_compliance_controller/stiffness/trans_z: 500 * /my_cartesian_compliance_controller/target_frame_topic: target_frame * /my_cartesian_compliance_controller/type: position_controll... * /my_cartesian_force_controller/end_effector_link: tool0 * /my_cartesian_force_controller/ft_sensor_ref_link: sensor_link * /my_cartesian_force_controller/joints: ['joint1', 'joint... * /my_cartesian_force_controller/pd_gains/rot_x/p: 0.01 * /my_cartesian_force_controller/pd_gains/rot_y/p: 0.01 * /my_cartesian_force_controller/pd_gains/rot_z/p: 0.01 * /my_cartesian_force_controller/pd_gains/trans_x/p: 0.05 * /my_cartesian_force_controller/pd_gains/trans_y/p: 0.05 * /my_cartesian_force_controller/pd_gains/trans_z/p: 0.05 * /my_cartesian_force_controller/robot_base_link: base_link * /my_cartesian_force_controller/type: position_controll... * /my_cartesian_motion_controller/end_effector_link: tool0 * /my_cartesian_motion_controller/joints: ['joint1', 'joint... * /my_cartesian_motion_controller/pd_gains/rot_x/p: 1.0 * /my_cartesian_motion_controller/pd_gains/rot_y/p: 1.0 * /my_cartesian_motion_controller/pd_gains/rot_z/p: 1.0 * /my_cartesian_motion_controller/pd_gains/trans_x/p: 10.0 * /my_cartesian_motion_controller/pd_gains/trans_y/p: 10.0 * /my_cartesian_motion_controller/pd_gains/trans_z/p: 10.0 * /my_cartesian_motion_controller/robot_base_link: base_link * /my_cartesian_motion_controller/target_frame_topic: target_frame * /my_cartesian_motion_controller/type: position_controll... * /my_joint_to_cartesian_controller/end_effector_link: tool0 * /my_joint_to_cartesian_controller/joint_limits/joint1/has_acceleration_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint1/has_velocity_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint1/max_acceleration: 10.0 * /my_joint_to_cartesian_controller/joint_limits/joint1/max_velocity: 3.15 * /my_joint_to_cartesian_controller/joint_limits/joint2/has_acceleration_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint2/has_velocity_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint2/max_acceleration: 10.0 * /my_joint_to_cartesian_controller/joint_limits/joint2/max_velocity: 3.15 * /my_joint_to_cartesian_controller/joint_limits/joint3/has_acceleration_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint3/has_velocity_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint3/max_acceleration: 10.0 * /my_joint_to_cartesian_controller/joint_limits/joint3/max_velocity: 3.15 * /my_joint_to_cartesian_controller/joint_limits/joint4/has_acceleration_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint4/has_velocity_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint4/max_acceleration: 10.0 * /my_joint_to_cartesian_controller/joint_limits/joint4/max_velocity: 3.2 * /my_joint_to_cartesian_controller/joint_limits/joint5/has_acceleration_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint5/has_velocity_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint5/max_acceleration: 10.0 * /my_joint_to_cartesian_controller/joint_limits/joint5/max_velocity: 3.2 * /my_joint_to_cartesian_controller/joint_limits/joint6/has_acceleration_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint6/has_velocity_limits: True * /my_joint_to_cartesian_controller/joint_limits/joint6/max_acceleration: 10.0 * /my_joint_to_cartesian_controller/joint_limits/joint6/max_velocity: 3.2 * /my_joint_to_cartesian_controller/joint_trajectory_controller/joints: ['joint1', 'joint... * /my_joint_to_cartesian_controller/joint_trajectory_controller/type: position_controll... * /my_joint_to_cartesian_controller/joints: ['joint1', 'joint... * /my_joint_to_cartesian_controller/robot_base_link: base_link * /my_joint_to_cartesian_controller/target_frame_topic: /my_cartesian_mot... * /my_joint_to_cartesian_controller/type: cartesian_control... * /my_motion_control_handle/end_effector_link: tool0 * /my_motion_control_handle/joints: ['joint1', 'joint... * /my_motion_control_handle/robot_base_link: base_link * /my_motion_control_handle/target_frame_topic: /my_cartesian_mot... * /my_motion_control_handle/type: cartesian_control... * /robot_description: responses = transport.receive_once() File "/opt/ros/noetic/lib/python3/dist-packages/rospy/impl/tcpros_base.py", line 737, in receive_once if __name__ == '__main__': main() File "/opt/ros/noetic/lib/controller_manager/spawner", line 190, in main resp = load_controller(name) File "/opt/ros/noetic/lib/python3/dist-packages/rospy/impl/tcpros_service.py", line 442, in __call__ return self.call(*args, **kwds) File "/opt/ros/noetic/lib/python3/dist-packages/rospy/impl/tcpros_service.py", line 532, in call self.stat_bytes += recv_buff(sock, b, p.buff_size) File "/opt/ros/noetic/lib/python3/dist-packages/rospy/impl/tcpros_base.py", line 109, in recv_buff raise TransportTerminated("unable to receive data from sender, check sender's logs for details") rospy.exceptions.TransportTerminated: unable to receive data from sender, check sender's logs for details During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/ros/noetic/lib/controller_manager/spawner", line 212, in raise ServiceException("transport error completing service call: %s"%(str(e))) rospy.service.ServiceException: transport error completing service call: unable to receive data from sender, check sender's logs for details if __name__ == '__main__': main() File "/opt/ros/noetic/lib/controller_manager/spawner", line 204, in main resp = switch_controller(loaded, [], 2, False, 0.0) File "/opt/ros/noetic/lib/python3/dist-packages/rospy/impl/tcpros_service.py", line 442, in __call__ return self.call(*args, **kwds) File "/opt/ros/noetic/lib/python3/dist-packages/rospy/impl/tcpros_service.py", line 532, in call raise ServiceException("transport error completing service call: %s"%(str(e))) rospy.service.ServiceException: transport error completing service call: unable to receive data from sender, check sender's logs for details [INFO] [1633938021.120907]: Shutting down spawner. Stopping and unloading controllers... [INFO] [1633938021.121749]: Stopping all controllers... [WARN] [1633938021.123698]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused [WARN] [1633938021.123920]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused [sim_hardware_interface-2] process has died [pid 6201, exit code -11, cmd /opt/ros/noetic/lib/ros_control_boilerplate/sim_hw_main __name:=sim_hardware_interface __log:=/home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/sim_hardware_interface-2.log]. log file: /home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/sim_hardware_interface-2*.log [ INFO] [1633938021.517795219]: Stereo is NOT SUPPORTED [ INFO] [1633938021.517855443]: OpenGL device: GeForce GTX 1060 3GB/PCIe/SSE2 [ INFO] [1633938021.517895800]: OpenGl version: 4,6 (GLSL 4,6). [joint_state_controller_spawner-4] process has died [pid 6203, exit code 1, cmd /opt/ros/noetic/lib/controller_manager/spawner joint_state_controller __name:=joint_state_controller_spawner __log:=/home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/joint_state_controller_spawner-4.log]. log file: /home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/joint_state_controller_spawner-4*.log [controller_spawner-5] process has died [pid 6204, exit code 1, cmd /opt/ros/noetic/lib/controller_manager/spawner --stopped my_cartesian_motion_controller my_cartesian_force_controller my_cartesian_compliance_controller my_motion_control_handle my_joint_to_cartesian_controller joint_trajectory_controller __name:=controller_spawner __log:=/home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/controller_spawner-5.log]. log file: /home/users/user.name/.ros/log/7c0ff3d4-2a66-11ec-bb0e-305a3a7a90fb/controller_spawner-5*.log [WARN] [1633938051.246333]: Controller Spawner couldn't find the expected controller_manager ROS interface. ```

stefanscherzinger commented 3 years ago

Hi @oruiz97

[ERROR] [1633938021.095060858]: MultiLibraryClassLoader: Could not create object of class type cartesian_controller_base::ForwardDynamicsSolver as no factory exists for it. Make sure that the library exists and was explicitly loaded through MultiLibraryClassLoader::loadLibrary()

This indicates that the controllers' solver (which is a separate library) cannot be loaded. Could you check the following?

1) After sourcing your devel/setup.bash, what's the output of

rospack plugins --attrib=plugin cartesian_controller_base

2) After navigating into devel/lib/, what's the output of

ldd -d libik_solvers.so

The workspace built correctly, right?

oruiz97 commented 3 years ago

Hi @stefanscherzinger, thanks for the help. Here's what I get

After sourcing your devel/setup.bash, what's the output of rospack plugins --attrib=plugin cartesian_controller_base

cartesian_controller_base /home/users/user.name/catkin_wscartesian/src/cartesian_controllers/cartesian_controller_base/ik_solver_plugin.xml

After navigating into devel/lib/, what's the output of ldd -d libik_solvers.so

    linux-vdso.so.1 (0x00007ffd967e4000)
    libkdl_parser.so => /opt/ros/noetic/lib/libkdl_parser.so (0x00007f19750ae000)
    liburdf.so => /opt/ros/noetic/lib/liburdf.so (0x00007f1975075000)
    liburdfdom_sensor.so.1.0 => /usr/lib/x86_64-linux-gnu/liburdfdom_sensor.so.1.0 (0x00007f1974feb000)
    liburdfdom_model_state.so.1.0 => /usr/lib/x86_64-linux-gnu/liburdfdom_model_state.so.1.0 (0x00007f1974fdf000)
    liburdfdom_model.so.1.0 => /usr/lib/x86_64-linux-gnu/liburdfdom_model.so.1.0 (0x00007f1974fba000)
    liburdfdom_world.so.1.0 => /usr/lib/x86_64-linux-gnu/liburdfdom_world.so.1.0 (0x00007f1974f93000)
    librosconsole_bridge.so => /opt/ros/noetic/lib/librosconsole_bridge.so (0x00007f1974f8c000)
    libcontrol_toolbox.so => /opt/ros/noetic/lib/libcontrol_toolbox.so (0x00007f1974f47000)
    libtinyxml.so.2.6.2 => /usr/lib/x86_64-linux-gnu/libtinyxml.so.2.6.2 (0x00007f1974d31000)
    librealtime_tools.so => /opt/ros/noetic/lib/librealtime_tools.so (0x00007f1974d29000)
    libroscpp.so => /opt/ros/noetic/lib/libroscpp.so (0x00007f1974b89000)
    libxmlrpcpp.so => /opt/ros/noetic/lib/libxmlrpcpp.so (0x00007f1974b64000)
    libeigen_conversions.so => /opt/ros/noetic/lib/libeigen_conversions.so (0x00007f1974b5d000)
    liborocos-kdl.so.1.4 => /usr/lib/liborocos-kdl.so.1.4 (0x00007f1974a90000)
    libdynamic_reconfigure_config_init_mutex.so => /opt/ros/noetic/lib/libdynamic_reconfigure_config_init_mutex.so (0x00007f1974a86000)
    libroscpp_serialization.so => /opt/ros/noetic/lib/libroscpp_serialization.so (0x00007f1974a81000)
    libclass_loader.so => /opt/ros/noetic/lib/libclass_loader.so (0x00007f1974a58000)
    libPocoFoundation.so.60 => /usr/lib/libPocoFoundation.so.60 (0x00007f197488a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1974885000)
    librosconsole.so => /opt/ros/noetic/lib/librosconsole.so (0x00007f1974832000)
    librosconsole_log4cxx.so => /opt/ros/noetic/lib/librosconsole_log4cxx.so (0x00007f1974814000)
    librosconsole_backend_interface.so => /opt/ros/noetic/lib/librosconsole_backend_interface.so (0x00007f197480f000)
    liblog4cxx.so.10 => /usr/lib/x86_64-linux-gnu/liblog4cxx.so.10 (0x00007f197462c000)
    libboost_regex.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.67.0 (0x00007f1974515000)
    librostime.so => /opt/ros/noetic/lib/librostime.so (0x00007f19744f2000)
    libcpp_common.so => /opt/ros/noetic/lib/libcpp_common.so (0x00007f19744e5000)
    libboost_thread.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.67.0 (0x00007f19744b9000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1974498000)
    libboost_chrono.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.67.0 (0x00007f197448d000)
    libboost_date_time.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.67.0 (0x00007f1974477000)
    libboost_atomic.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.67.0 (0x00007f1974472000)
    libconsole_bridge.so.0.4 => /usr/lib/x86_64-linux-gnu/libconsole_bridge.so.0.4 (0x00007f197446c000)
    libroslib.so => /opt/ros/noetic/lib/libroslib.so (0x00007f1974456000)
    librospack.so => /opt/ros/noetic/lib/librospack.so (0x00007f1974410000)
    libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007f1973f13000)
    libboost_filesystem.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.67.0 (0x00007f1973ef3000)
    libboost_program_options.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.67.0 (0x00007f1973e6c000)
    libboost_system.so.1.67.0 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.67.0 (0x00007f1973e65000)
    libtinyxml2.so.6 => /usr/lib/x86_64-linux-gnu/libtinyxml2.so.6 (0x00007f1973e4d000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1973cc9000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1973b46000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1973b2a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1973969000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1975137000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f197395d000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f19738e9000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f19736cb000)
    libapr-1.so.0 => /usr/lib/x86_64-linux-gnu/libapr-1.so.0 (0x00007f1973692000)
    libaprutil-1.so.0 => /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0 (0x00007f1973664000)
    libicudata.so.63 => /usr/lib/x86_64-linux-gnu/libicudata.so.63 (0x00007f1971c74000)
    libicui18n.so.63 => /usr/lib/x86_64-linux-gnu/libicui18n.so.63 (0x00007f1971997000)
    libicuuc.so.63 => /usr/lib/x86_64-linux-gnu/libicuuc.so.63 (0x00007f19717c8000)
    libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f197178b000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f1971751000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f197174a000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f1971741000)

The workspace built correctly, right?

Yes, no errors show up when building

stefanscherzinger commented 3 years ago

@oruiz97 That is somewhat unfortunate. I can't spot anything unusual. If the issue still persists, could you switch-on debug logging for the sim_hardware_interface? Maybe the whole process of loading the libraries gives more insight. I think this can be activated in rqt with Plugins -> Logging -> Logger level.

Mohatashem commented 3 years ago

Hello, I also tried launching the examples.launch file exactly as done by @oruiz97. I am using Ubuntu 18.04 and ROS-Melodic.

The terminal has the following output:

... logging to /home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/roslaunch-pp1744-Precision-5550-9747.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

xacro: in-order processing became default in ROS Melodic. You can drop the option.
started roslaunch server http://pp1744-Precision-5550:43843/

SUMMARY
========

PARAMETERS
 * /generic_hw_control_loop/cycle_time_error_threshold: 0.03
 * /generic_hw_control_loop/loop_hz: 100
 * /hardware_interface/joints: ['joint1', 'joint...
 * /hardware_interface/sim_control_mode: 0
 * /joint_state_controller/publish_rate: 50
 * /joint_state_controller/type: joint_state_contr...
 * /joint_trajectory_controller/joints: ['joint1', 'joint...
 * /joint_trajectory_controller/type: position_controll...
 * /my_cartesian_compliance_controller/compliance_ref_link: tool0
 * /my_cartesian_compliance_controller/end_effector_link: tool0
 * /my_cartesian_compliance_controller/ft_sensor_ref_link: sensor_link
 * /my_cartesian_compliance_controller/joints: ['joint1', 'joint...
 * /my_cartesian_compliance_controller/pd_gains/rot_x/p: 0.01
 * /my_cartesian_compliance_controller/pd_gains/rot_y/p: 0.01
 * /my_cartesian_compliance_controller/pd_gains/rot_z/p: 0.01
 * /my_cartesian_compliance_controller/pd_gains/trans_x/p: 0.05
 * /my_cartesian_compliance_controller/pd_gains/trans_y/p: 0.05
 * /my_cartesian_compliance_controller/pd_gains/trans_z/p: 0.05
 * /my_cartesian_compliance_controller/robot_base_link: base_link
 * /my_cartesian_compliance_controller/stiffness/rot_x: 100
 * /my_cartesian_compliance_controller/stiffness/rot_y: 100
 * /my_cartesian_compliance_controller/stiffness/rot_z: 100
 * /my_cartesian_compliance_controller/stiffness/trans_x: 500
 * /my_cartesian_compliance_controller/stiffness/trans_y: 500
 * /my_cartesian_compliance_controller/stiffness/trans_z: 500
 * /my_cartesian_compliance_controller/target_frame_topic: target_frame
 * /my_cartesian_compliance_controller/type: position_controll...
 * /my_cartesian_force_controller/end_effector_link: tool0
 * /my_cartesian_force_controller/ft_sensor_ref_link: sensor_link
 * /my_cartesian_force_controller/joints: ['joint1', 'joint...
 * /my_cartesian_force_controller/pd_gains/rot_x/p: 0.01
 * /my_cartesian_force_controller/pd_gains/rot_y/p: 0.01
 * /my_cartesian_force_controller/pd_gains/rot_z/p: 0.01
 * /my_cartesian_force_controller/pd_gains/trans_x/p: 0.05
 * /my_cartesian_force_controller/pd_gains/trans_y/p: 0.05
 * /my_cartesian_force_controller/pd_gains/trans_z/p: 0.05
 * /my_cartesian_force_controller/robot_base_link: base_link
 * /my_cartesian_force_controller/type: position_controll...
 * /my_cartesian_motion_controller/end_effector_link: tool0
 * /my_cartesian_motion_controller/joints: ['joint1', 'joint...
 * /my_cartesian_motion_controller/pd_gains/rot_x/p: 1.0
 * /my_cartesian_motion_controller/pd_gains/rot_y/p: 1.0
 * /my_cartesian_motion_controller/pd_gains/rot_z/p: 1.0
 * /my_cartesian_motion_controller/pd_gains/trans_x/p: 10.0
 * /my_cartesian_motion_controller/pd_gains/trans_y/p: 10.0
 * /my_cartesian_motion_controller/pd_gains/trans_z/p: 10.0
 * /my_cartesian_motion_controller/robot_base_link: base_link
 * /my_cartesian_motion_controller/target_frame_topic: target_frame
 * /my_cartesian_motion_controller/type: position_controll...
 * /my_joint_to_cartesian_controller/end_effector_link: tool0
 * /my_joint_to_cartesian_controller/joint_limits/joint1/has_acceleration_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint1/has_velocity_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint1/max_acceleration: 10.0
 * /my_joint_to_cartesian_controller/joint_limits/joint1/max_velocity: 3.15
 * /my_joint_to_cartesian_controller/joint_limits/joint2/has_acceleration_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint2/has_velocity_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint2/max_acceleration: 10.0
 * /my_joint_to_cartesian_controller/joint_limits/joint2/max_velocity: 3.15
 * /my_joint_to_cartesian_controller/joint_limits/joint3/has_acceleration_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint3/has_velocity_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint3/max_acceleration: 10.0
 * /my_joint_to_cartesian_controller/joint_limits/joint3/max_velocity: 3.15
 * /my_joint_to_cartesian_controller/joint_limits/joint4/has_acceleration_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint4/has_velocity_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint4/max_acceleration: 10.0
 * /my_joint_to_cartesian_controller/joint_limits/joint4/max_velocity: 3.2
 * /my_joint_to_cartesian_controller/joint_limits/joint5/has_acceleration_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint5/has_velocity_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint5/max_acceleration: 10.0
 * /my_joint_to_cartesian_controller/joint_limits/joint5/max_velocity: 3.2
 * /my_joint_to_cartesian_controller/joint_limits/joint6/has_acceleration_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint6/has_velocity_limits: True
 * /my_joint_to_cartesian_controller/joint_limits/joint6/max_acceleration: 10.0
 * /my_joint_to_cartesian_controller/joint_limits/joint6/max_velocity: 3.2
 * /my_joint_to_cartesian_controller/joint_trajectory_controller/joints: ['joint1', 'joint...
 * /my_joint_to_cartesian_controller/joint_trajectory_controller/type: position_controll...
 * /my_joint_to_cartesian_controller/joints: ['joint1', 'joint...
 * /my_joint_to_cartesian_controller/robot_base_link: base_link
 * /my_joint_to_cartesian_controller/target_frame_topic: /my_cartesian_mot...
 * /my_joint_to_cartesian_controller/type: cartesian_control...
 * /my_motion_control_handle/end_effector_link: tool0
 * /my_motion_control_handle/joints: ['joint1', 'joint...
 * /my_motion_control_handle/robot_base_link: base_link
 * /my_motion_control_handle/target_frame_topic: /my_cartesian_mot...
 * /my_motion_control_handle/type: cartesian_control...
 * /robot_description: <?xml version="1....
 * /robot_state_publisher/publish_frequency: 50.0
 * /robot_state_publisher/tf_prefix: 
 * /rosdistro: melodic
 * /rosversion: 1.14.12

NODES
  /
    controller_spawner (controller_manager/spawner)
    joint_state_controller_spawner (controller_manager/spawner)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    rviz (rviz/rviz)
    sim_hardware_interface (ros_control_boilerplate/sim_hw_main)
  /my_joint_to_cartesian_controller/
    controller_spawner (controller_manager/spawner)

ROS_MASTER_URI=http://localhost:11311

process[sim_hardware_interface-1]: started with pid [9796]
process[robot_state_publisher-2]: started with pid [9797]
process[joint_state_controller_spawner-3]: started with pid [9798]
[ INFO] [1636718264.036318464]: Waiting for model URDF on the ROS param server at location: //robot_description
process[controller_spawner-4]: started with pid [9807]
process[my_joint_to_cartesian_controller/controller_spawner-5]: started with pid [9812]
process[rviz-6]: started with pid [9815]
[ INFO] [1636718264.139226947]: GenericHWInterface Ready.
[ INFO] [1636718264.139264262]: SimHWInterface Ready.
[ INFO] [1636718264.159699929]: rviz version 1.13.21
[ INFO] [1636718264.159749740]: compiled against Qt version 5.9.5
[ INFO] [1636718264.159763390]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1636718264.168712710]: Forcing OpenGl version 0.
[ INFO] [1636718264.312768871]: Stereo is NOT SUPPORTED
[ INFO] [1636718264.312823164]: OpenGL device: Mesa DRI Intel(R) UHD Graphics P630 (CML GT2)
[ INFO] [1636718264.312849636]: OpenGl version: 3.0 (GLSL 1.3).
[INFO] [1636718264.382230]: Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [1636718264.383940]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1636718264.385595]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1636718264.387525]: Loading controller: joint_state_controller
[INFO] [1636718264.393035]: Controller Spawner: Loaded controllers: joint_state_controller
[INFO] [1636718264.403133]: Started controllers: joint_state_controller
Traceback (most recent call last):
  File "/opt/ros/melodic/lib/controller_manager/spawner", line 212, in <module>
    if __name__ == '__main__': main()
  File "/opt/ros/melodic/lib/controller_manager/spawner", line 190, in main
    resp = load_controller(name)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 442, in __call__
    return self.call(*args, **kwds)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 532, in call
    raise ServiceException("transport error completing service call: %s"%(str(e)))
rospy.service.ServiceException: transport error completing service call: unable to receive data from sender, check sender's logs for details
[WARN] [1636718264.559785]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[sim_hardware_interface-1] process has died [pid 9796, exit code -11, cmd /opt/ros/melodic/lib/ros_control_boilerplate/sim_hw_main __name:=sim_hardware_interface __log:=/home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/sim_hardware_interface-1.log].
log file: /home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/sim_hardware_interface-1*.log
[controller_spawner-4] process has died [pid 9807, exit code 1, cmd /opt/ros/melodic/lib/controller_manager/spawner --stopped my_cartesian_motion_controller my_cartesian_force_controller my_cartesian_compliance_controller my_motion_control_handle my_joint_to_cartesian_controller joint_trajectory_controller __name:=controller_spawner __log:=/home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/controller_spawner-4.log].
log file: /home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/controller_spawner-4*.log
[rviz-6] process has finished cleanly
log file: /home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/rviz-6*.log
[WARN] [1636718294.659502]: Controller Spawner couldn't find the expected controller_manager ROS interface.
[my_joint_to_cartesian_controller/controller_spawner-5] process has finished cleanly
log file: /home/pp1744/.ros/log/f93e2b80-4390-11ec-83f3-d03c1fd5e0d5/my_joint_to_cartesian_controller-controller_spawner-5*.log

The output for rospack plugins --attrib=plugin cartesian_controller_base is cartesian_controller_base /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/ik_solver_plugin.xml

The output for ldd -d libik_solvers.so is

linux-vdso.so.1 (0x00007ffd78286000)
        libroscpp.so => /opt/ros/melodic/lib/libroscpp.so (0x00007f5ea5674000)
        liborocos-kdl.so.1.4 => /opt/ros/melodic/lib/liborocos-kdl.so.1.4 (0x00007f5ea53df000)
        libdynamic_reconfigure_config_init_mutex.so => /opt/ros/melodic/lib/libdynamic_reconfigure_config_init_mutex.so (0x00007f5ea51d7000)
        libroscpp_serialization.so => /opt/ros/melodic/lib/libroscpp_serialization.so (0x00007f5ea4fd4000)
        libclass_loader.so => /opt/ros/melodic/lib/libclass_loader.so (0x00007f5ea4dad000)
        librosconsole.so => /opt/ros/melodic/lib/librosconsole.so (0x00007f5ea4b77000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5ea4958000)
        libconsole_bridge.so.0.4 => /usr/lib/x86_64-linux-gnu/libconsole_bridge.so.0.4 (0x00007f5ea4753000)
        libboost_system.so.1.65.1 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 (0x00007f5ea454e000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5ea41c5000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5ea3e27000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5ea3c0f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5ea381e000)
        libxmlrpcpp.so => /opt/ros/melodic/lib/libxmlrpcpp.so (0x00007f5ea35fe000)
        librostime.so => /opt/ros/melodic/lib/librostime.so (0x00007f5ea33de000)
        libcpp_common.so => /opt/ros/melodic/lib/libcpp_common.so (0x00007f5ea31d3000)
        libboost_thread.so.1.65.1 => /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1 (0x00007f5ea2fae000)
        libboost_chrono.so.1.65.1 => /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.65.1 (0x00007f5ea2da9000)
        libboost_filesystem.so.1.65.1 => /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1 (0x00007f5ea2b8f000)
        libPocoFoundation.so.50 => /usr/lib/libPocoFoundation.so.50 (0x00007f5ea27e6000)
        librosconsole_log4cxx.so => /opt/ros/melodic/lib/librosconsole_log4cxx.so (0x00007f5ea25cb000)
        librosconsole_backend_interface.so => /opt/ros/melodic/lib/librosconsole_backend_interface.so (0x00007f5ea23c9000)
        liblog4cxx.so.10 => /usr/lib/x86_64-linux-gnu/liblog4cxx.so.10 (0x00007f5ea2000000)
        libboost_regex.so.1.65.1 => /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.65.1 (0x00007f5ea1cf8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5ea618e000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5ea1af0000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5ea18ec000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f5ea167a000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5ea145d000)
        libapr-1.so.0 => /usr/lib/x86_64-linux-gnu/libapr-1.so.0 (0x00007f5ea1228000)
        libaprutil-1.so.0 => /usr/lib/x86_64-linux-gnu/libaprutil-1.so.0 (0x00007f5ea0ffd000)
        libicui18n.so.60 => /usr/lib/x86_64-linux-gnu/libicui18n.so.60 (0x00007f5ea0b5c000)
        libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f5ea07a4000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f5ea059d000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f5ea0365000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f5ea0133000)
        libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f5e9e58a000)

On trying to swtich on the debug-logging for sim_hardware_interface in rqt the 'Loggers' and 'Level' sections are blank, while the terminal displays the following:

During get_service_class_by_name "/sim_hardware_interface/get_loggers":
Unable to communicate with service [/sim_hardware_interface/get_loggers], address [rosrpc://pp1744-Precision-5550:36991]

I am not sure how to proceed from here. Any ideas/suggestions are appreciated. Thank you.

stefanscherzinger commented 3 years ago

@Mohatashem

[sim_hardware_interface-1] process has died

I guess at that point we need to go into debugging this thing. If you find some time, could you do the following?

Build your workspace with debug symbols:
```
catkin_make -DCMAKE_BUILD_TYPE=Debug
```
Create a .gdbinit file in your home folder, if not already existent, and add the following:
```
set pagination off
```
This is helpful when spawning the node in gdb later so that you don't have to hurry before the controller manager times out.
Start the examples launch with debugging enabled:
```
roslaunch cartesian_controller_examples examples.launch debug:=True
```
This will start the sim_hardware_interface in gdb inside a screen session. It personally like that approach to pick-up the screen session in a terminal of my choice.
In some terminal, call
```
screen -r
```
This will get you inside gdb. When the program crashes, type backtrace. Could you post that output here?

Mohatashem commented 3 years ago

Hello, Thank you for the response. Here is the output of from screen -r followed by backtrace when the program crashes:

GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/ros/melodic/lib/ros_control_boilerplate/sim_hw_main...(no debugging symbols found)...done.
/home/pp1744/.ros/log: Is a directory.
Starting program: /opt/ros/melodic/lib/ros_control_boilerplate/sim_hw_main __name:=sim_hardware_interface __log:=/home/pp1744/.ros/log/72015566-46c4-11ec-ae27-d03c1fd5e0d5/sim_hardware_interface-2.log
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffed969700 (LWP 9821)]
[New Thread 0x7fffed168700 (LWP 9822)]
[New Thread 0x7fffec967700 (LWP 9823)]
[New Thread 0x7fffe7fff700 (LWP 9824)]
[New Thread 0x7fffe77fe700 (LWP 9825)]
[New Thread 0x7fffe6ffd700 (LWP 9826)]
[New Thread 0x7fffe67fc700 (LWP 9827)]
[ INFO] [1637057009.704165332]: Waiting for model URDF on the ROS param server at location: //robot_description
[ INFO] [1637057009.806918663]: GenericHWInterface Ready.
[ INFO] [1637057009.806970376]: SimHWInterface Ready.
[New Thread 0x7fffe57ac700 (LWP 9839)]

Thread 7 "sim_hw_main" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6ffd700 (LWP 9826)]
0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1

(gdb) backtrace
#0  0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1
#1  0x00007fffe4d114d0 in boost::filesystem::exists (p=...) at /usr/local/include/boost/filesystem/operations.hpp:404
#2  0x00007fffe4d45d3a in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::getPackageFromPluginXMLFilePath (this=0x7fffd8002a70, plugin_xml_file_path="/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/ik_solver_plugin.xml") at /opt/ros/melodic/include/pluginlib/././class_loader_imp.hpp:515
#3  0x00007fffe4d3e7a2 in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::processSingleXMLPluginFile (this=0x7fffd8002a70, xml_file="/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/ik_solver_plugin.xml", classes_available=std::map with 0 elements) at /opt/ros/melodic/include/pluginlib/././class_loader_imp.hpp:671
#4  0x00007fffe4d39e3a in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::determineAvailableClasses (this=0x7fffd8002a70, plugin_xml_paths=std::vector of length 1, capacity 1 = {...}) at /opt/ros/melodic/include/pluginlib/././class_loader_imp.hpp:261
#5  0x00007fffe4d367b8 in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::ClassLoader (this=0x7fffd8002a70, package="cartesian_controller_base", base_class="cartesian_controller_base::IKSolver", attrib_name="plugin", plugin_xml_paths=std::vector of length 0, capacity 0) at /opt/ros/melodic/include/pluginlib/././class_loader_imp.hpp:92
#6  0x00007fffe4d2f33b in cartesian_controller_base::CartesianControllerBase<hardware_interface::PositionJointInterface>::init (this=0x7fffd8006eb0, hw=0x55555578a880, nh=...) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79
#7  0x00007fffe4d2e04e in cartesian_motion_controller::CartesianMotionController<hardware_interface::PositionJointInterface>::init (this=0x7fffd8006db0, hw=0x55555578a880, nh=...) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_motion_controller/include/cartesian_motion_controller/cartesian_motion_controller.hpp:63
#8  0x00007fffe4d2eae1 in controller_interface::Controller<hardware_interface::PositionJointInterface>::initRequest (this=0x7fffd8006eb0, robot_hw=0x55555578a640, root_nh=..., controller_nh=..., claimed_resources=std::vector of length 0, capacity 0) at /opt/ros/melodic/include/controller_interface/controller.h:119
#9  0x00007ffff61c911e in controller_manager::ControllerManager::loadController(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#10 0x00007ffff61ca453 in controller_manager::ControllerManager::loadControllerSrv(controller_manager_msgs::LoadControllerRequest_<std::allocator<void> >&, controller_manager_msgs::LoadControllerResponse_<std::allocator<void> >&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#11 0x00007ffff61dd8a6 in ros::ServiceCallbackHelperT<ros::ServiceSpec<controller_manager_msgs::LoadControllerRequest_<std::allocator<void> >, controller_manager_msgs::LoadControllerResponse_<std::allocator<void> > > >::call(ros::ServiceCallbackHelperCallParams&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#12 0x00007ffff76a2974 in ros::ServiceCallback::call() () from /opt/ros/melodic/lib/libroscpp.so
#13 0x00007ffff76f6829 in ros::CallbackQueue::callOneCB(ros::CallbackQueue::TLS*) () from /opt/ros/melodic/lib/libroscpp.so
#14 0x00007ffff76f7b15 in ros::CallbackQueue::callOne(ros::WallDuration) () from /opt/ros/melodic/lib/libroscpp.so
#15 0x00007ffff774ef44 in ros::AsyncSpinnerImpl::threadFunc() () from /opt/ros/melodic/lib/libroscpp.so
#16 0x00007ffff5733bcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#17 0x00007ffff53056db in start_thread (arg=0x7fffe6ffd700) at pthread_create.c:463
#18 0x00007ffff6b9771f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

stefanscherzinger commented 3 years ago

@Mohatashem Thanks for providing the detailed output and thanks for investing your time!

That is really weird. It is still successful in constructing a boost::filesystem::path with the parent folder of the ik_solver_plugin.xml file (see code here, scroll to line 524).

However, it later fails in checking if the package.xml file in that same directory exists. But instead of just returning false, it segfaults. That's really weird.

Could you check what's going on in the following directory?

ls -l /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/

And get the sha1 checksum of the package.xml file?

sha1sum package.xml

Mine (master freshly downloaded) is: 163a1ad0fd56228d2e772077dd166d10bb354fb3 package.xml

Edit: On what commit are you?

git rev-parse HEAD

Mohatashem commented 3 years ago

@stefanscherzinger no worries, It's a pleasure! The output to

ls -l /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/

is

total 36
drwxrwxr-x 2 pp1744 pp1744 4096 Nov 12 09:08 cfg
-rw-rw-r-- 1 pp1744 pp1744 8507 Nov 12 09:08 CMakeLists.txt
-rw-rw-r-- 1 pp1744 pp1744 1119 Nov 12 09:08 ik_solver_plugin.xml
drwxrwxr-x 3 pp1744 pp1744 4096 Nov 12 09:08 include
-rw-rw-r-- 1 pp1744 pp1744 2662 Nov 12 09:08 package.xml
-rw-rw-r-- 1 pp1744 pp1744   85 Nov 12 09:08 README.md
drwxrwxr-x 2 pp1744 pp1744 4096 Nov 12 09:08 src

My cheksum is the same as yours:

sha1sum package.xml
163a1ad0fd56228d2e772077dd166d10bb354fb3  package.xml

The commit is master

stefanscherzinger commented 3 years ago

Thanks @Mohatashem

I'm really not sure what's happening here. Could you switch to the replace-boost-with-std branch, completely remove build and devel folders and rebuild? I replaced boost functionality with the standard in the hope of removing possible pitfalls with user-side Boost versions.

If that's not solving the issue, I need to find a way to reproduce this myself on Melodic for further debugging.

stefanscherzinger commented 3 years ago

@Mohatashem Did you make progress on this?

Edit: Please use the latest master branch.

Mohatashem commented 3 years ago

@stefanscherzinger Pardon me for the delayed reply. I had tried with the replace-boost-with-std a few days ago but couldn't get to reply to you. But nothing changes. I even tried the steps you mentioned for debugging and compared the terminal outputs to the older ones. the errors stay the same. I still can't seem to figure out what's wrong.

The same holds true with the latest master branch. I just tried it as well.

One additional thing, I am not sure it if it is related, but in every try so far, as the Rviz starts, the RobotModel also throws an error. See attached screenshot: rviz_cartesian_ctrl_screenshot

stefanscherzinger commented 3 years ago

@Mohatashem Thanks for the fast reply! Before continuing, could you confirm that you completely deleted both your devel and build folder?

Mohatashem commented 3 years ago

@stefanscherzinger Yes, of course, that's what i did before rebuilding.

stefanscherzinger commented 3 years ago

@Mohatashem Alright, thanks for clarifying this.

Your remark on RViz is interesting. ros_control_boilerplate should at least come up with the joint_state_controller, which should publish the joint_states for the robot_state_publisher to complete the TF tree. And the model should look normal in RViz. Maybe that points us to something general not working.

Could you outcomment the controller spawner in the launch file? Something like

        <!-- Spawn controllers -->
        <!--
        <node name="controller_spawner" pkg="controller_manager" type="spawner"
                args="- -stopped
                my_cartesian_motion_controller
                my_cartesian_force_controller
                my_cartesian_compliance_controller
                my_motion_control_handle
                my_joint_to_cartesian_controller
                joint_trajectory_controller
                "
        />
        -->

Note the whitespace in - - stopped to keep valid xml syntax.

Don't run it in debug mode for this. Use the normal mode. Let's see if this minimal setup without the cartesian_controllers works.

stefanscherzinger commented 3 years ago

If that doesn't change anything in RViz, which I assume, maybe some further dependencies are missing. Could you run this in the terminal where you normally build the workspace

rosdep install --from-paths src --ignore-src -y -s

and paste the output here? The -s flag only simulates the install instead of actually doing it. Then we can check if something is missing.

Mohatashem commented 3 years ago

@stefanscherzinger

Could you outcomment the controller spawner in the launch file? Something like

Yes, when I comment it out, the RobotModel loads fine: Screenshot from 2021-11-27 14-39-18

And the terminal output now does show that the joint_state_controller is loaded. This is the last part of the shell:

[INFO] [1638020333.531156]: Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [1638020333.533340]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1638020333.535472]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1638020333.537105]: Loading controller: joint_state_controller
[INFO] [1638020333.543902]: Controller Spawner: Loaded controllers: joint_state_controller
[INFO] [1638020333.553778]: Started controllers: joint_state_controller
[WARN] [1638020363.808553]: Controller Spawner couldn't find the expected controller_manager ROS interface.
[my_joint_to_cartesian_controller/controller_spawner-5] process has finished cleanly
log file: /home/pp1744/.ros/log/5bca0c22-4f87-11ec-b03d-d03c1fd5e0d5/my_joint_to_cartesian_controller-controller_spawner-5*.log

I have not yet ran rosdep install --from-paths src --ignore-src -y -s as Rviz model seemed ok, should I still run it?

stefanscherzinger commented 3 years ago

@Mohatashem

[my_joint_to_cartesian_controller/controller_spawner-5] process has finished cleanly

That shouldn't load. Make sure that you do not spawn any of the controllers from the cartesian_controllers package.

But it's interesting that RViz shows the model fine.

I have not yet ran rosdep install --from-paths src --ignore-src -y -s as Rviz model seemed ok, should I still run it?

Yes, please. Let's see what that says.

Mohatashem commented 3 years ago

@stefanscherzinger

That shouldn't load. Make sure that you do not spawn any of the controllers.

Okay. Just to clarify, I had commented out only this i.e. the whole node tag. Now I also commented out the this i.e. everything within the group tag. The model in rviz loads fine I left this one uncommented because without it the Rviz model does not load as before.

Could you run this in the terminal where you normally build the workspace

rosdep install --from-paths src --ignore-src -y -s

No output returned, I wonder if I am doing anything wrong
pp1744@pp1744-Precision-5550:~/rosworkspaces/cartesian_control_test_ws$ rosdep install --from-paths src --ignore-src -y -s
pp1744@pp1744-Precision-5550:~/rosworkspaces/cartesian_control_test_ws$ 

stefanscherzinger commented 3 years ago

@Mohatashem

Now I also commented out the this i.e. everything within the group tag. The model in rviz loads fine I left this one uncommented because without it the Rviz model does not load as before.

Yes, thanks, that's right.

No output returned, I wonder if I am doing anything wrong

That's good. That means that all dependencies are already installed.

Could you now launch that stuff again with debug:=True (leaving the parts outcommented) And open rqt with the controller manager plugin?

Screenshot from 2021-11-27 18-59-32

After loading the my_cartesian_motion_controller (richt klick -> load), could you again post the output you get from the backtrace in screen from gdb?

Let's see if that again segfaults somewhere in pluginlib.

Mohatashem commented 3 years ago

@stefanscherzinger

Here is the output:


GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/ros/melodic/lib/ros_control_boilerplate/sim_hw_main...(no debugging symbols found)...done.
/home/pp1744/.ros/my_debug_log: No such file or directory.
Starting program: /opt/ros/melodic/lib/ros_control_boilerplate/sim_hw_main __name:=sim_hardware_interface __log:=/home/pp1744/.ros/log/79228c48-4fb6-11ec-a349-d03c1fd5e0d5/sim_hardware_interface-2.log
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffed969700 (LWP 10906)]
[New Thread 0x7fffed168700 (LWP 10907)]
[New Thread 0x7fffec967700 (LWP 10908)]
[New Thread 0x7fffe7fff700 (LWP 10909)]
[New Thread 0x7fffe77fe700 (LWP 10910)]
[New Thread 0x7fffe6ffd700 (LWP 10911)]
[New Thread 0x7fffe67fc700 (LWP 10912)]
[ INFO] [1638040569.157151426]: Waiting for model URDF on the ROS param server at location: //robot_description
[ INFO] [1638040569.260168766]: GenericHWInterface Ready.
[ INFO] [1638040569.260199875]: SimHWInterface Ready.
[New Thread 0x7fffe57ac700 (LWP 10946)]

Thread 7 "sim_hw_main" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe6ffd700 (LWP 10911)]
0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1

(gdb) backtrace
#0  0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1
#1  0x00007fffe4d114d0 in boost::filesystem::exists(boost::filesystem::path const&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#2  0x00007fffe4d45d3a in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::getPackageFromPluginXMLFilePath(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#3  0x00007fffe4d3e7a2 in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::processSingleXMLPluginFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, pluginlib::ClassDesc, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, pluginlib::ClassDesc> > >&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#4  0x00007fffe4d39e3a in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::determineAvailableClasses(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#5  0x00007fffe4d367b8 in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::ClassLoader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#6  0x00007fffe4d2f33b in cartesian_controller_base::CartesianControllerBase<hardware_interface::PositionJointInterface>::init(hardware_interface::PositionJointInterface*, ros::NodeHandle&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#7  0x00007fffe4d2e04e in cartesian_motion_controller::CartesianMotionController<hardware_interface::PositionJointInterface>::init(hardware_interface::PositionJointInterface*, ros::NodeHandle&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#8  0x00007fffe4d2eae1 in controller_interface::Controller<hardware_interface::PositionJointInterface>::initRequest(hardware_interface::RobotHW*, ros::NodeHandle&, ros::NodeHandle&, std::vector<hardware_interface::InterfaceResources, std::allocator<hardware_interface::InterfaceResources> >&) () from /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib//libcartesian_motion_controller.so
#9  0x00007ffff61c911e in controller_manager::ControllerManager::loadController(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#10 0x00007ffff61ca453 in controller_manager::ControllerManager::loadControllerSrv(controller_manager_msgs::LoadControllerRequest_<std::allocator<void> >&, controller_manager_msgs::LoadControllerResponse_<std::allocator<void> >&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#11 0x00007ffff61dd8a6 in ros::ServiceCallbackHelperT<ros::ServiceSpec<controller_manager_msgs::LoadControllerRequest_<std::allocator<void> >, controller_manager_msgs::LoadControllerResponse_<std::allocator<void> > > >::call(ros::ServiceCallbackHelperCallParams&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#12 0x00007ffff76a2974 in ros::ServiceCallback::call() () from /opt/ros/melodic/lib/libroscpp.so
#13 0x00007ffff76f6829 in ros::CallbackQueue::callOneCB(ros::CallbackQueue::TLS*) () from /opt/ros/melodic/lib/libroscpp.so
#14 0x00007ffff76f7b15 in ros::CallbackQueue::callOne(ros::WallDuration) () from /opt/ros/melodic/lib/libroscpp.so
#15 0x00007ffff774ef44 in ros::AsyncSpinnerImpl::threadFunc() () from /opt/ros/melodic/lib/libroscpp.so
#16 0x00007ffff5733bcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#17 0x00007ffff53056db in start_thread (arg=0x7fffe6ffd700) at pthread_create.c:463
#18 0x00007ffff6b9771f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

Thread 7 "sim_hw_main" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffe6ffd700 (LWP 10911)] 0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1

This appears exactly when I right click->load my_cartesian_motion_controller

stefanscherzinger commented 3 years ago

@Mohatashem

Thanks again for going through all this! It again points to some corruption from frame #2 downwards. If I were to provoke this myself, I would build and link pluginlib as a shared object to the boost system dependencies and then change the boost version to run into some ABI incompatibilities.. But on Melodic, pluginlib hasn't changed since February last year and Boost 1.65.1 seems to be the default for Bionic... weird.

As a last thing to try just in case, could you get pluginlib and build that inside the ROS workspace with the cartesian_controllers from source?

Build with debug symbols enabled:

catkin_make -DCMAKE_BUILD_TYPE=Debug

Does that reproduce the same segfault when loading the my_cartesian_motion_controller?

Edit: I checked the CI for Melodic and that successfully uses pluginlib version 1.12.1, which differs from the recent version 1.12.2. Before building pluginlib from source, could you check your version with

rosversion pluginlib

Mohatashem commented 3 years ago

@stefanscherzinger On my system the pluginlib version is 1.12.1

Again just to be sure, I am doing everything correctly, this is what I did:

Cloned the pluginlib (same version) to src folder of my workspace.
deleted the build and devel folders.
ran catkin_make -DCMAKE_BUILD_TYPE=Debug

Kindly, let me know did I miss anything in the steps, so far?

Moving on,

I repeated what we have done so far. I received the same segfault just as I load the my_cartesian_motion_controller:


Thread 6 "sim_hw_main" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe77fe700 (LWP 30349)]
0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1
(gdb) backtrace
#0  0x00007ffff50ee6cd in boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*) () from /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1
#1  0x00007fffe4d114e0 in boost::filesystem::exists (p=...) at /usr/local/include/boost/filesystem/operations.hpp:404
#2  0x00007fffe4d45d4a in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::getPackageFromPluginXMLFilePath (this=0x7fffd4006ce0, plugin_xml_file_path="/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/ik_solver_plugin.xml") at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:515
#3  0x00007fffe4d3e7b2 in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::processSingleXMLPluginFile (this=0x7fffd4006ce0, xml_file="/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/ik_solver_plugin.xml", classes_available=std::map with 0 elements) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:671
#4  0x00007fffe4d39e4a in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::determineAvailableClasses (this=0x7fffd4006ce0, plugin_xml_paths=std::vector of length 1, capacity 1 = {...}) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:261
#5  0x00007fffe4d367c8 in pluginlib::ClassLoader<cartesian_controller_base::IKSolver>::ClassLoader (this=0x7fffd4006ce0, package="cartesian_controller_base", base_class="cartesian_controller_base::IKSolver", attrib_name="plugin", plugin_xml_paths=std::vector of length 0, capacity 0) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:92
#6  0x00007fffe4d2f34b in cartesian_controller_base::CartesianControllerBase<hardware_interface::PositionJointInterface>::init (this=0x7fffd400b0f0, hw=0x55555578a9c0, nh=...) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79
#7  0x00007fffe4d2e05e in cartesian_motion_controller::CartesianMotionController<hardware_interface::PositionJointInterface>::init (this=0x7fffd400aff0, hw=0x55555578a9c0, nh=...) at /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_motion_controller/include/cartesian_motion_controller/cartesian_motion_controller.hpp:63
#8  0x00007fffe4d2eaf1 in controller_interface::Controller<hardware_interface::PositionJointInterface>::initRequest (this=0x7fffd400b0f0, robot_hw=0x55555578a780, root_nh=..., controller_nh=..., claimed_resources=std::vector of length 0, capacity 0) at /opt/ros/melodic/include/controller_interface/controller.h:119
#9  0x00007ffff61c911e in controller_manager::ControllerManager::loadController(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#10 0x00007ffff61ca453 in controller_manager::ControllerManager::loadControllerSrv(controller_manager_msgs::LoadControllerRequest_<std::allocator<void> >&, controller_manager_msgs::LoadControllerResponse_<std::allocator<void> >&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#11 0x00007ffff61dd8a6 in ros::ServiceCallbackHelperT<ros::ServiceSpec<controller_manager_msgs::LoadControllerRequest_<std::allocator<void> >, controller_manager_msgs::LoadControllerResponse_<std::allocator<void> > > >::call(ros::ServiceCallbackHelperCallParams&) () from /opt/ros/melodic/lib/libcontroller_manager.so
#12 0x00007ffff76a2974 in ros::ServiceCallback::call() () from /opt/ros/melodic/lib/libroscpp.so
#13 0x00007ffff76f6829 in ros::CallbackQueue::callOneCB(ros::CallbackQueue::TLS*) () from /opt/ros/melodic/lib/libroscpp.so
#14 0x00007ffff76f7b15 in ros::CallbackQueue::callOne(ros::WallDuration) () from /opt/ros/melodic/lib/libroscpp.so
#15 0x00007ffff774ef44 in ros::AsyncSpinnerImpl::threadFunc() () from /opt/ros/melodic/lib/libroscpp.so
#16 0x00007ffff5733bcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#17 0x00007ffff53056db in start_thread (arg=0x7fffe77fe700) at pthread_create.c:463
#18 0x00007ffff6b9771f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I even repeated the steps with pluginlib version 1.12.2 (after removing the previous version from the workspace) and got the same error.

I wonder what to do next. One more thing, when running catkin_make with or without degub symbols, there are some warnings. Wonder whether they are useful:

[ 77%] Building CXX object cartesian_controllers/joint_to_cartesian_controller/CMakeFiles/joint_to_cartesian_controller.dir/src/joint_to_cartesian_controller.cpp.o
[ 81%] Building CXX object cartesian_controllers/joint_to_cartesian_controller/CMakeFiles/joint_to_cartesian_controller.dir/src/JointControllerAdapter.cpp.o
In file included from /opt/ros/melodic/include/ros/assert.h:35:0,
                 from /opt/ros/melodic/include/ros/common.h:36,
                 from /opt/ros/melodic/include/ros/publisher.h:32,
                 from /opt/ros/melodic/include/ros/node_handle.h:32,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.h:44,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_motion_controller/include/cartesian_motion_controller/cartesian_motion_controller.h:44,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_motion_controller/src/cartesian_motion_controller.cpp:44:
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp: In instantiation of ‘pluginlib::ClassLoader<T>::ClassLoader(std::__cxx11::string, std::__cxx11::string, std::__cxx11::string, std::vector<std::__cxx11::basic_string<char> >) [with T = cartesian_controller_base::IKSolver; std::__cxx11::string = std::__cxx11::basic_string<char>]’:
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79:25:   required from ‘bool cartesian_controller_base::CartesianControllerBase<HardwareInterface>::init(HardwareInterface*, ros::NodeHandle&) [with HardwareInterface = hardware_interface::PositionJointInterface]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_motion_controller/src/cartesian_motion_controller.cpp:65:1:   required from here
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:83:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader", "Creating ClassLoader, base = %s, address = %p",
   ^
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:93:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader",
   ^
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp: In instantiation of ‘pluginlib::ClassLoader<T>::~ClassLoader() [with T = cartesian_controller_base::IKSolver]’:
/usr/local/include/boost/core/checked_delete.hpp:34:5:   required from ‘void boost::checked_delete(T*) [with T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/detail/shared_count.hpp:134:34:   required from ‘boost::detail::shared_count::shared_count(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:271:20:   required from ‘void boost::detail::sp_pointer_construct(boost::shared_ptr<X>*, Y*, boost::detail::shared_count&) [with T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:349:44:   required from ‘boost::shared_ptr<T>::shared_ptr(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:620:9:   required from ‘void boost::shared_ptr<T>::reset(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79:24:   required from ‘bool cartesian_controller_base::CartesianControllerBase<HardwareInterface>::init(HardwareInterface*, ros::NodeHandle&) [with HardwareInterface = hardware_interface::PositionJointInterface]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_motion_controller/src/cartesian_motion_controller.cpp:65:1:   required from here
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:102:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader", "Destroying ClassLoader, base = %s, address = %p",
   ^
In file included from /opt/ros/melodic/include/ros/assert.h:35:0,
                 from /opt/ros/melodic/include/ros/common.h:36,
                 from /opt/ros/melodic/include/ros/publisher.h:32,
                 from /opt/ros/melodic/include/ros/node_handle.h:32,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.h:44,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_force_controller/include/cartesian_force_controller/cartesian_force_controller.h:44,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_force_controller/src/cartesian_force_controller.cpp:44:
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp: In instantiation of ‘pluginlib::ClassLoader<T>::ClassLoader(std::__cxx11::string, std::__cxx11::string, std::__cxx11::string, std::vector<std::__cxx11::basic_string<char> >) [with T = cartesian_controller_base::IKSolver; std::__cxx11::string = std::__cxx11::basic_string<char>]’:
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79:25:   required from ‘bool cartesian_controller_base::CartesianControllerBase<HardwareInterface>::init(HardwareInterface*, ros::NodeHandle&) [with HardwareInterface = hardware_interface::PositionJointInterface]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_force_controller/src/cartesian_force_controller.cpp:65:1:   required from here
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:83:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader", "Creating ClassLoader, base = %s, address = %p",
   ^
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:93:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader",
   ^
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp: In instantiation of ‘pluginlib::ClassLoader<T>::~ClassLoader() [with T = cartesian_controller_base::IKSolver]’:
/usr/local/include/boost/core/checked_delete.hpp:34:5:   required from ‘void boost::checked_delete(T*) [with T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/detail/shared_count.hpp:134:34:   required from ‘boost::detail::shared_count::shared_count(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:271:20:   required from ‘void boost::detail::sp_pointer_construct(boost::shared_ptr<X>*, Y*, boost::detail::shared_count&) [with T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:349:44:   required from ‘boost::shared_ptr<T>::shared_ptr(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:620:9:   required from ‘void boost::shared_ptr<T>::reset(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79:24:   required from ‘bool cartesian_controller_base::CartesianControllerBase<HardwareInterface>::init(HardwareInterface*, ros::NodeHandle&) [with HardwareInterface = hardware_interface::PositionJointInterface]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_force_controller/src/cartesian_force_controller.cpp:65:1:   required from here
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:102:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader", "Destroying ClassLoader, base = %s, address = %p",
   ^
[ 85%] Linking CXX shared library /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib/libjoint_to_cartesian_controller.so
[ 85%] Built target joint_to_cartesian_controller
[ 88%] Linking CXX shared library /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib/libcartesian_motion_controller.so
[ 88%] Built target cartesian_motion_controller
[ 92%] Linking CXX shared library /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib/libcartesian_force_controller.so
[ 92%] Built target cartesian_force_controller
Scanning dependencies of target cartesian_compliance_controller
[ 96%] Building CXX object cartesian_controllers/cartesian_compliance_controller/CMakeFiles/cartesian_compliance_controller.dir/src/cartesian_compliance_controller.cpp.o
In file included from /opt/ros/melodic/include/ros/assert.h:35:0,
                 from /opt/ros/melodic/include/ros/common.h:36,
                 from /opt/ros/melodic/include/ros/publisher.h:32,
                 from /opt/ros/melodic/include/ros/node_handle.h:32,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.h:44,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_compliance_controller/include/cartesian_compliance_controller/cartesian_compliance_controller.h:44,
                 from /home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_compliance_controller/src/cartesian_compliance_controller.cpp:44:
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp: In instantiation of ‘pluginlib::ClassLoader<T>::ClassLoader(std::__cxx11::string, std::__cxx11::string, std::__cxx11::string, std::vector<std::__cxx11::basic_string<char> >) [with T = cartesian_controller_base::IKSolver; std::__cxx11::string = std::__cxx11::basic_string<char>]’:
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79:25:   required from ‘bool cartesian_controller_base::CartesianControllerBase<HardwareInterface>::init(HardwareInterface*, ros::NodeHandle&) [with HardwareInterface = hardware_interface::PositionJointInterface]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_compliance_controller/src/cartesian_compliance_controller.cpp:65:1:   required from here
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:83:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader", "Creating ClassLoader, base = %s, address = %p",
   ^
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:93:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader",
   ^
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp: In instantiation of ‘pluginlib::ClassLoader<T>::~ClassLoader() [with T = cartesian_controller_base::IKSolver]’:
/usr/local/include/boost/core/checked_delete.hpp:34:5:   required from ‘void boost::checked_delete(T*) [with T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/detail/shared_count.hpp:134:34:   required from ‘boost::detail::shared_count::shared_count(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:271:20:   required from ‘void boost::detail::sp_pointer_construct(boost::shared_ptr<X>*, Y*, boost::detail::shared_count&) [with T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:349:44:   required from ‘boost::shared_ptr<T>::shared_ptr(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/usr/local/include/boost/smart_ptr/shared_ptr.hpp:620:9:   required from ‘void boost::shared_ptr<T>::reset(Y*) [with Y = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>; T = pluginlib::ClassLoader<cartesian_controller_base::IKSolver>]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_controller_base/include/cartesian_controller_base/cartesian_controller_base.hpp:79:24:   required from ‘bool cartesian_controller_base::CartesianControllerBase<HardwareInterface>::init(HardwareInterface*, ros::NodeHandle&) [with HardwareInterface = hardware_interface::PositionJointInterface]’
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/cartesian_controllers/cartesian_compliance_controller/src/cartesian_compliance_controller.cpp:65:1:   required from here
/opt/ros/melodic/include/ros/console.h:348:26: warning: format ‘%p’ expects argument of type ‘void*’, but argument 9 has type ‘pluginlib::ClassLoader<cartesian_controller_base::IKSolver>*’ [-Wformat=]
     ::ros::console::print(filter, __rosconsole_define_location__loc.logger_, __rosconsole_define_location__loc.level_, __FILE__, __LINE__, __ROSCONSOLE_FUNCTION__, __VA_ARGS__)
     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:351:5: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER’
     ROSCONSOLE_PRINT_AT_LOCATION_WITH_FILTER(NULL, __VA_ARGS__)
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:387:7: note: in expansion of macro ‘ROSCONSOLE_PRINT_AT_LOCATION’
       ROSCONSOLE_PRINT_AT_LOCATION(__VA_ARGS__); \
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/melodic/include/ros/console.h:572:35: note: in expansion of macro ‘ROS_LOG_COND’
 #define ROS_LOG(level, name, ...) ROS_LOG_COND(true, level, name, __VA_ARGS__)
                                   ^~~~~~~~~~~~
/opt/ros/melodic/include/rosconsole/macros_generated.h:60:36: note: in expansion of macro ‘ROS_LOG’
 #define ROS_DEBUG_NAMED(name, ...) ROS_LOG(::ros::console::levels::Debug, std::string(ROSCONSOLE_NAME_PREFIX) + "." + name, __VA_ARGS__)
                                    ^~~~~~~
/home/pp1744/rosworkspaces/cartesian_control_test_ws/src/pluginlib/include/pluginlib/././class_loader_imp.hpp:102:3: note: in expansion of macro ‘ROS_DEBUG_NAMED’
   ROS_DEBUG_NAMED("pluginlib.ClassLoader", "Destroying ClassLoader, base = %s, address = %p",
   ^
[100%] Linking CXX shared library /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib/libcartesian_compliance_controller.so
[100%] Built target cartesian_compliance_controller

Edit: Forgot to add that, also when trying to load the my_cartesian_motion_controller, the controller manageralso shuts down with the following:

Traceback (most recent call last):
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rqt_controller_manager/controller_manager.py", line 287, in _on_ctrl_menu
    self._load_controller(ctrl.name)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rqt_controller_manager/controller_manager.py", line 331, in _load_controller
    self._load_srv.call(LoadControllerRequest(name=name))
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 532, in call
    raise ServiceException("transport error completing service call: %s"%(str(e)))
rospy.service.ServiceException: transport error completing service call: unable to receive data from sender, check sender's logs for details
Aborted (core dumped)

stefanscherzinger commented 3 years ago

@Mohatashem Thanks for all the information.

Kindly, let me know did I miss anything in the steps, so far?

No, that was alright. I wanted to test exactly what you did. The idea was to see if we had ABI inconsistencies. They would've gone away by relinking pluginlib against the system Boost dependency. But that's not it. Unfortunately I'm running out of ideas to guide you remotely through this thing without wasting too much of your time.

there are some warnings. Wonder whether they are useful:

I wouldn't think so, no. What keeps irritating me is this in the backtrace (frame #3):

ik_solver_plugin.xml", classes_available=std::map with 0 elements)

I would expect 4 classes instead.

I'll try if I can reproduce the whole thing somehow on Melodic myself and let you know.

stefanscherzinger commented 3 years ago

@Mohatashem

There's one thing you could do. Could you provide the CoreDump file of the segfault? That would allow us to have a look at the exact memory state when the program crashed.

That would be like (I hope this works on 18.04):

Restore the examples.launch file, i.e. start everything like from a clean git clone.
Also remove pluginlib from the src folder
Remove build and devel
Build your workspace with debug symbols
In the terminal where you call roslaunch, call ulimit -c unlimited. This will allow to create core files
Launch everything in that terminal and wait for the segfault
End all ROS-related processes
Navigate to /var/crash and find the .crash file of the segfault
Unpack that to a temporary folder with apport-unpack your-crash-file tmp_folder
Attach the CoreDump file from the tmp_folder here to this post.

I can then debug with gdb using that core file.

Mohatashem commented 2 years ago

@stefanscherzinger

Unfortunately I'm running out of ideas to guide you remotely through this thing without wasting too much of your time.

No problem at all. I do understand it and I thank you so much for your time and help. I really appreciate it. As a last attempt, I have attached the file. Here is it CoreDump.zip

stefanscherzinger commented 2 years ago

@Mohatashem

Thanks, but that's unfortunately not the right one. Was there some other ..crash file? Something with ros_control_boilerplate_sim_hw_main or similar in its name?

We want to find the core dump for this executable:

Thread 6 "sim_hw_main" received signal SIGSEGV, Segmentation fault.

You can check the core file with the file command. The attachment gives me:

file CoreDump
CoreDump: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'python /opt/ros/melodic/lib/rqt_controller_manager/rqt_controller_manager', real uid: 1000, effective uid: 1000, real gid: 1000, effecti
ve gid: 1000, execfn: '/usr/bin/python', platform: 'x86_64'

Additionally, could you also append the libcartesian_motion_controller.so? It should be in devel/lib/. I will be needing that for the symbols. They are normally not included in the core file.

Edit: The core file might also be located under /var/lib/apport/coredump.

Mohatashem commented 2 years ago

@stefanscherzinger Sorry for the delayed reply. For some reason, I don't seem to be getting a crash file with ros_control_boilerplate_sim_hw_main name. I started anew and everything. However I realized that in my latest try Thread 6 "sim_hw_main" received signal SIGSEGV, Segmentation fault. does not appear in the launch terminal, it only appears in the terminal of screen -r (where we do the backtrace). Even then, I couldn't find any such crash file (my /var/crash and /var/lib/apport/coredump) are both empty.

I will retry it again maybe later or tomorrow and get will surely get back to you if it is alright?

One other thing, I would like to mention is that I cloned the repo on two other systems with melodic installation and there it worked flawlessly, which is weird for me, given I usually test everything on this system before installing dependencies (if required) on the other system and deploying.

stefanscherzinger commented 2 years ago

@Mohatashem

it only appears in the terminal of screen -r (where we do the backtrace).

Oh, right. I forgot to mention. Please launch it normally for creating the core file. No debugging. Remember to set ulimit -c unlimited in the terminal where you launch everything. A SIGSEGV triggers a core dump. At least it just did on my 20.04 when sending a kill -11 to the process of my running sim_hw_main. gdb seems to swallow that.

will surely get back to you if it is alright?

Yes, I would like to get to the bottom of this specific problem. It might also be relevant for other people.

I would like to mention is that I cloned the repo on two other systems with melodic installation and there it worked flawlessly

That's good to hear! It also agrees with the integration test for Melodic of this repo.

Mohatashem commented 2 years ago

@stefanscherzinger I am really sorry, for some reason, I still cannot see the SIGSEGV triggering a core dump. Both the folders you mentioned in var are empty. Let me list the steps I followed just in case I skipped something:

Starting a new, I git cloned everything to src
I built the workspace using catkin_make -DCMAKE_BUILD_TYPE=Debug
Sourced the workspace source devel/setup.bash
Called ulimit -c unlimited in a terminal
roslaunch cartesian_controller_examples examples.launch in the same terminal as step 4.

I do see that [sim_hardware_interface-1] process from ros_control_boilerplate dies, but I cannot see core dump.

I kill the terminal to close the ROS related processes.

But there is something interesting. Here is the 'error' part from the terminal where the launch file was launched:

Traceback (most recent call last):
  File "/opt/ros/melodic/lib/controller_manager/spawner", line 212, in <module>
    if __name__ == '__main__': main()
  File "/opt/ros/melodic/lib/controller_manager/spawner", line 190, in main
    resp = load_controller(name)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 442, in __call__
    return self.call(*args, **kwds)
  File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 532, in call
    raise ServiceException("transport error completing service call: %s"%(str(e)))
rospy.service.ServiceException: transport error completing service call: unable to receive data from sender, check sender's logs for details
[WARN] [1638525840.318197]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[sim_hardware_interface-1] process has died [pid 15915, exit code -11, cmd /opt/ros/melodic/lib/ros_control_boilerplate/sim_hw_main my_motion_control_handle/target_frame:=target_frame my_cartesian_motion_controller/target_frame:=target_frame my_cartesian_compliance_controller/target_frame:=target_frame my_cartesian_force_controller/target_wrench:=target_wrench my_cartesian_compliance_controller/target_wrench:=target_wrench __name:=sim_hardware_interface __log:=/home/pp1744/.ros/log/b0c7df60-541e-11ec-8b92-d03c1fd5e0d5/sim_hardware_interface-1.log].
log file: /home/pp1744/.ros/log/b0c7df60-541e-11ec-8b92-d03c1fd5e0d5/sim_hardware_interface-1*.log
[controller_spawner-4] process has died [pid 15918, exit code 1, cmd /opt/ros/melodic/lib/controller_manager/spawner --stopped my_cartesian_motion_controller my_cartesian_force_controller my_cartesian_compliance_controller my_motion_control_handle my_joint_to_cartesian_controller joint_trajectory_controller __name:=controller_spawner __log:=/home/pp1744/.ros/log/b0c7df60-541e-11ec-8b92-d03c1fd5e0d5/controller_spawner-4.log].
log file: /home/pp1744/.ros/log/b0c7df60-541e-11ec-8b92-d03c1fd5e0d5/controller_spawner-4*.log
Qt: Session management error: Could not open network socket
[ INFO] [1638525841.041048974]: rviz version 1.13.21
[ INFO] [1638525841.041084862]: compiled against Qt version 5.9.5
[ INFO] [1638525841.041094114]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1638525841.048061593]: Forcing OpenGl version 0.
[ INFO] [1638525841.202446029]: Stereo is NOT SUPPORTED
[ INFO] [1638525841.202521639]: OpenGL device: Mesa DRI Intel(R) UHD Graphics P630 (CML GT2)
[ INFO] [1638525841.202550339]: OpenGl version: 3.0 (GLSL 1.3).

In the above messages from the terminal, it mentions log file: /home/pp1744/.ros/log/b0c7df60-541e-11ec-8b92-d03c1fd5e0d5/sim_hardware_interface-1*.log for the sim_hardware_interface but when I navigate to that folder, this file is not present. I tried it a few times. Every other log file mentioned is present in that folder. Let me know if there is anything else you want me to do.

Edit: PFA libcartesian_motion_controller.zip

stefanscherzinger commented 2 years ago

@Mohatashem

Ok, let's take a different approach for creating the core file. We can do that also in gdb (I just found that out). Try to follow these steps:

Build your workspace with debug symbols as before
Launch the examples.launch file with debug:=True
Pick-up the screen session as before and wait for the segfault
In gdb, check the current directory with pwd. That's were the core file will be saved to
In gdb, call gcore. This will create a core file
Exit gdb
Get the core file and upload it here.

When doing that on my system, the core files from gdb end up in ~/.ros in my home directory.

Edit: PFA libcartesian_motion_controller.zip

Thanks for uploading the symbol file!

Mohatashem commented 2 years ago

@stefanscherzinger Thank you so much! Please find attached. If possible also kindly share any resources/link where you learnt how to debug in this manner. I am fairly new to all this core.zip

libcartesian_motion_controller.zip

stefanscherzinger commented 2 years ago

@Mohatashem

I had a look at the core file and there's something that might indicate a clash with two Boost installations.

Using your core file, I did the following:

Setup a fresh Ubuntu 18.04 in VirtualBox
Installed ROS (ros-melodic-desktop-full)
Loaded the symbols of libcartesian_motion_controller.so and the core-file
Checked that gdb found the relevant symbols

Step 4. included to put libcartesian_motion_controller.so into /home/pp1744/rosworkspaces/cartesian_control_test_ws/devel/lib, which I created to mimic your setup.

Going through the stack frames, I noticed that your setup uses /usr/local/include/boost/filesystem/operations.hpp, whereas a freshly installed Ubuntu 18.04 with Melodic uses /usr/include/boost.... This indicates that you have a second, non-default Boost installation under /usr/local/include. However, this include is linked against the default Boost library /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1 instead of against the local library (probably somewhere in /usr/local/lib).

To test whether this is in fact the problem, you could try two things:

Use the Ubuntu 18.04 default Boost installation
Or consistently use your second Boost installation (but don't mix them)

This answer in the ROS wiki explains how to do that.

For enforcing the default installation (1.), it might be easiest to remove your additional Boost installation under /usr/local.

For enforcing your additional installation (2.), you could probably remove build and devel and rebuild with

catkin_make -DBOOST_ROOT=/usr/local -DBoost_NO_SYSTEM_PATHS=ON

Mohatashem commented 2 years ago

@stefanscherzinger Yes, indeed. I first tried 2., but the I had the same issues. Then following suggestion 1.I simply removed the additional installation. And now it works fine. I have no idea how the additional installation got there in the first place. I really appreciate the time you took for helping me out on this. You've my deepest gratitude. Thank you so much!

stefanscherzinger commented 2 years ago

@Mohatashem You are welcome! And thanks again for hanging in there for so long and your continuous interest in getting this fixed!

fzi-forschungszentrum-informatik / cartesian_controllers

Cannot run the examples launch file #29