RVIZ shutting down after adding 5th sample

tlpss commented 2 years ago

we are trying to use moveit_calibration for eye2hand but RVIZ keeps shutting down as soon as we enter the 5th sample (after which the solver is executed). However there is no logging information to provide a clue on what is going wrong.

This is the trace for RVIZ:

roslaunch ur3e_moveit_config moveit_rviz.launch config:=true
... logging to /home/tlips/.ros/log/4a34bf84-844d-11ec-8add-8597f1441aa0/roslaunch-idlab293-25631.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://idlab293:33883/

SUMMARY
========

PARAMETERS
 * /rosdistro: noetic
 * /rosversion: 1.15.14

NODES
  /
    rviz_idlab293_25631_3275010300154187535 (rviz/rviz)

ROS_MASTER_URI=http://localhost:11311

process[rviz_idlab293_25631_3275010300154187535-1]: started with pid [25652]
[ INFO] [1643822863.062593710]: rviz version 1.14.13
[ INFO] [1643822863.062640676]: compiled against Qt version 5.12.8
[ INFO] [1643822863.062648675]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1643822863.068210393]: Forcing OpenGl version 0.
[ INFO] [1643822863.418079370]: Stereo is NOT SUPPORTED
[ INFO] [1643822863.418136912]: OpenGL device: NVIDIA GeForce MX450/PCIe/SSE2
[ INFO] [1643822863.418151286]: OpenGl version: 4,6 (GLSL 4,6).
[ INFO] [1643822904.548447105]: Loading robot model 'ur3e_robot'...
[ WARN] [1643822904.561107681]: Kinematics solver doesn't support #attempts anymore, but only a timeout.
Please remove the parameter '/robot_description_kinematics/manipulator/kinematics_solver_attempts' from your configuration.
[ INFO] [1643822904.609348619]: Starting planning scene monitor
[ INFO] [1643822904.610620314]: Listening to '/move_group/monitored_planning_scene'
[ INFO] [1643822904.817469465]: No active joints or end effectors found for group 'endeffector'. Make sure that kinematics.yaml is loaded in this node's namespace.
[ INFO] [1643822904.817893470]: No active joints or end effectors found for group 'endeffector'. Make sure that kinematics.yaml is loaded in this node's namespace.
[ INFO] [1643822904.821706450]: Constructing new MoveGroup connection for group 'endeffector' in namespace ''
[ INFO] [1643822905.995890641]: Ready to take commands for planning group endeffector.
[ INFO] [1643822915.049221823]: No active joints or end effectors found for group 'endeffector'. Make sure that kinematics.yaml is loaded in this node's namespace.
[ INFO] [1643822942.859288709]: TFVisualTools Ready.
[ INFO] [1643822942.862212362]: Loading robot model 'ur3e_robot'...
[ INFO] [1643822942.885327721]: Loading robot model 'ur3e_robot'...
[ INFO] [1643822942.909825415]: Loading robot model 'ur3e_robot'...
[ INFO] [1643822942.935208431]: Loading robot model 'ur3e_robot'...
[ INFO] [1643822942.972592477]: Loading robot model 'ur3e_robot'...
[ INFO] [1643822943.029287553]: Starting planning scene monitor
[ INFO] [1643822943.030217538]: Listening to '/move_group/monitored_planning_scene'
[ INFO] [1643822943.033008529]: Loading robot model 'ur3e_robot'...
[ INFO] [1643822943.073819001]: Ready to take commands for planning group endeffector.
[ INFO] [1643822943.074640168]: handeye calibration gui created.
[ERROR] [1643822949.156258981]: CameraInfo msg is NULL.
[ERROR] [1643822950.554681461]: CameraInfo msg is NULL.
[ WARN] [1643822963.783794140]: Topic '/rviz_visual_tools' unable to connect to any subscribers within 0.5 sec. It is possible initially published visual messages will be lost.
[ INFO] [1643822991.145941657]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643822993.161990484]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643822995.177637381]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643822997.194112324]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643822999.210080480]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823001.226658951]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823003.242095746]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823005.258376971]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823007.273910359]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823009.289833944]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823011.306189738]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823013.322193626]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823015.338165648]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823016.704828059]: Ready to take commands for planning group manipulator.
[ INFO] [1643823017.353678456]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823019.357962458]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[ INFO] [1643823021.381845480]: Set target real dimensions: 
marker_measured_size 0,200000
marker_measured_separation 0,020000

[rviz_idlab293_25631_3275010300154187535-1] process has died [pid 25652, exit code -11, cmd /opt/ros/noetic/lib/rviz/rviz __name:=rviz_idlab293_25631_3275010300154187535 __log:=/home/tlips/.ros/log/4a34bf84-844d-11ec-8add-8597f1441aa0/rviz_idlab293_25631_3275010300154187535-1.log].
log file: /home/tlips/.ros/log/4a34bf84-844d-11ec-8add-8597f1441aa0/rviz_idlab293_25631_3275010300154187535-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done

We use a ZED2i camera with the provided ros wrapper. The robot is a UR3e, which is controlled using the Universal Robotics ROS drivers.

Ubuntu 20.04 is used with Noetic. We build moveit_calibration from source after installing moveit using apt.

AexMm commented 2 years ago

Did you by any chance solve that problem? I am using a hc10 and a zedm on ubuntu 20.04, python3.8.10, noetic and i am having the same problem.

Apparently the error happens in handeye_solver_default.cpp near line 30x "python_value = PyEval_CallObject(python_func_add_sample, python_args_sample[i]);" (at least that's what my line-by-line-print-debug-attempt gave me ...)

tlpss commented 2 years ago

Hi @AexMm I found the same the same line caused the issue. I built all the dependencies from source (https://github.com/crigroup/handeye and https://github.com/crigroup/baldor) and found that the shutdown was caused by a segmentation fault in the BLAS library somehow for me.

the call stack (from the line you quote) at the moment of the segmentation fault is:

https://github.com/crigroup/handeye/blob/452a1cc272f62c213e038e513d2bd72a418532a0/src/handeye/calibrator.py#L72
https://github.com/crigroup/baldor/blob/cdc58aa941e170108458c885cc481576204d108f/src/baldor/transform.py#L70 after which the numpy.dot operation crashed.

Running the operation with the same arguments in my CLI worked just fine, so I was a little clueless about what was going on.

At that point I abandoned my search for now (since I wanted to get it running in ROS2 anyways).

Feel free to let me know if you were able to find the issue.

AexMm commented 2 years ago

Hello again @tlpss. my path also led me to the numpy.dot() problem and running the same stuff on the CLI seemed way more cooperative that the handeye_calibration part.

When i removed libopenblas* (0.3.8) from my system to install it manually (0.3.9) like this comment (https://github.com/mlpack/mlpack/issues/2437#issue-628160038) suggested, i found the handeye calibration working again.

So apparently openblas is the troublemaker in this case. I cannot confirm if the update to 0.3.9 really helps, but i can confirm the trace to openblas.

tlpss commented 2 years ago

@AexMm , thanks for your update! I'll try this in the next days and will keep you posted.

tlpss commented 2 years ago

When i removed libopenblas* (0.3.8) from my system to install it manually (0.3.9) like this comment (https://github.com/mlpack/mlpack/issues/2437#issue-628160038) suggested, i found the handeye calibration working again.

@AexMm, thanks for your fix. Changing the version of libopenblas did the trick for us as well.

Mohamedgalil commented 1 year ago

Thanks @AexMm for the fix. It also works for me. Here are the steps that I followed to uninstall existing OpenBlas and reinstall it from the source for version 0.3.23:

# uncomment the next line to remove existing openblas. Do this with caution!! 
# sudo apt -y remove libopenblas* 

# download openblas from official github page
wget https://github.com/xianyi/OpenBLAS/releases/download/v0.3.23/OpenBLAS-0.3.23.tar.gz
tar xf OpenBLAS-0.3.23.tar.gz 
cd OpenBLAS-0.3.23/
# Make then install
make
make install

moveit / moveit_calibration

RVIZ shutting down after adding 5th sample #109