calibrate.launch, tracking_base and tracking_object frame

[mbajlo]

Hello,

I am working with the variety

In the readme.md it is explained to make handeye transform I should:

1. frames

create a new handeye_calibrate.launch file, which includes the robot's and tracking system's launch files, as well as easy_handeye's calibrate.launch as illustrated below in the next section "Calibration" in each of your launch files where you need the result of the calibration, include easy_handeye's publish.launch as illustrated below in the section "Publishing"

• Calibration For both use cases, you can either launch the calibrate.launch launch file, or you can include it in another launchfile as shown below.

    <!-- fill in the following parameters according to your robot's published tf frames -->
    <arg name="robot_base_frame" value="/base_link"/>
    <arg name="robot_effector_frame" value="/ee_link"/>

In the robot_base_frame and in the robot_effector_frame I should add my robots base frame and end effector frame, these are published by the robot_state_publisher under /tf. This is clear to me, but the part with tracking_base_frame and tracking_marker_frame

    <!-- fill in the following parameters according to your tracking system's published tf frames -->
    <arg name="tracking_base_frame" value="/optical_origin"/>
    <arg name="tracking_marker_frame" value="/optical_target"/>

I do not understand where are these coming from? Should I have some tracking node which is subscribed to the camera topic and it is publishing these frames under /tf like the robot frames?

2. rqt_easy_handeye package is used to move the robot automatically or by hand. and in each position calibration will make the calculation/estimation?

3. Publishing publish.launch is publishing transformation in the /tf what transform? Transform of end effector frame in the camera frame? if it is end effetor in the camera frame, what is this used for? to see the transform in each step? and how it is getting more and more accurate with each new pose?

[marcoesposito1988]

Hello @mbajlo,

1. yes, you should have a tracking software running, that is able to publish on /tf the transformation between the camera optical frame and an object of interest. A typical example are AR markers, which are designed so that it is possible to compute this transformation based on the image (and the intrinsic calibration matrix of the camera). You can use the ArUco library for this, which was integrated into OpenCV. To this end, you can use my package easy_aruco.

You should

• perform the intrinsic calibration of the RGB camera that you are using (if you are using an RGB-D camera such as a kinect, they usually have an acceptable calibration from the factory that is automatically picked up by the ROS driver). If you use the ROS camera_calibration package, after saving the intrinsic calibration this will be automatically used by the rest of the ROS packages.
• generate and print an AR marker, or even better a ChArUco board if you are using a tracking software that supports it (there are generators online, and I included one into easy_aruco, which supports ChArUco boards)
• measure with a ruler the size of the AR marker after printing (most times the printer auto scales the image, so you can't rely on what you pass to the generator)
• write a launch file that launches your camera driver and the tracking software, specifying the camera video topic, the marker ID and size
• verify that the tracking is working on its own (you should see the transform in tf, and also in rviz if you turn on the tf plugin)
• copy the content of the launch file into your calibrate.launch. In the end, your calibrate.launch should start the robot, start the video camera, start the tracking software, include easy_handeye/calibrate.launch. If something is not working, check the connections between the topics and that you are passing the right arguments to each launch file that you are including.

2. rqt_easy_handeye contains a UI for sampling the transformations from tf, and a separate UI to move the robot around the initial position. The latter is optional, you can also move the robot by hand around. You need to acquire multiple samples, rotating the end effector in all directions (x,y,z). After you moved the robot in one direction, you need to wait until everything in rviz has stopped moving (because of eventual lag), and then you can acquire a sample; then continue, until you have enough samples.

3. the publisher will transform only your calibration transformation: the transformation between the robot base and the camera frame in case of an "eye on base" calibration, or between the robot end effector and the camera for an "eye in hand" calibration. This is the transformation that you don't know exactly, and that you want to find out with the calibration procedure.

The more samples you acquire, the better your calibration (unless you introduce outliers into the samples). For more details, please refer to the Tsai-Lenz algorithm.

[mbajlo]

Hello @marcoesposito1988,

First of all, thank for the reply. I will take a look into the easy_arucopackage to publish the transform between the camera and the marker. The idea here was to follow the VISP camera calibration (intrinsic and extrinsic) and after that to make PBVS and IBVS. I am trying to understand this because I will try it with a lot of different robots in a simulation. But I need to follow a procedure like I have a real robot, because after simulation I will do the same thing with the actual robot. But just for my understanding, I have few questions:

• I was following the VISP tutorials for the intrinsic camera calibration, it generates a file camera.xml does the camera_calibration package generates the same thing? what did you mean by this part

after saving the intrinsic calibration this will be automatically used by the rest of the ROS packages.

The intrinsic parameters are saved in some file? how the rest of the ROS packages know where is this file and which other ROS packages should use the intrinsic calibration parametes?

• After extrinsic calibration I would be able to continue with the PBVS and IBVS from the VISP?

• To measure the ArUco marker I need to measure it sides(painted in black) not the size of the one square?

• On the internet I have seen a lot of pictures of the different kinds of markers/chessboards. But I am confused with the Z-axis (blue axis in the rviz). (positive Z-axis is a cross product of the X-positive direction*Y-positive direction) The positive direction of the Z-axis of the marker frame should look outside of the marker and towards to the camera frame? It shouldn't look into the marker? This part confuses me because it matters in which direction it is looking to? Otherwise the calibration will not work? Where the positive direction of the Z-axis is looking in your easy_handeye package?

EDIT: 25.3.2020 I tried charuco tracker. The question is, does the orientation of the XYZ of the charuco board is relevant, this refers to the question from before, does the Z-axis has to look towards the camera? From what I have seen, when I move the charuco board in Y direction, in rviz it is moving in X direction, this is not ok, or maybe I am doing something wrong. I keep getting this warning:

[ WARN] [1585174483.910026617]: The input topic '/fanuc_1_eye_on_hand/camera_object' is not yet advertised [ WARN] [1585174483.921283663]: The input topic '/fanuc_1_eye_on_hand/world_effector' is not yet advertised

Where this topics should be advertised from?

Thanks

[mbajlo]

Hello @marcoesposito1988,

I have figured it out, thanks ;)