charuco board

[assia855]

Hi @cviviers, could you please share the Charuco board you use in create_dataset I'm struggling with mine. Even after changing the detect_Charuco_pose_board parameters from number_markers_x = 8, number_markers_y = 5 to number_markers_x = 4, number_markers_y = 3, it only computes the pose for a few frames in my dataset. Additionally, when I test the poses of the frames I obtained in Sanity check function, I get these kinds of poses.

[cviviers]

Hi @assia855,

Some general tips:

• the size of the printed markers need to exactly match the size you have them in your charuco description. In my example everything is in meters. So a marker needs to be 0.04m or 4 cm.
• Using Gimp you can print to a specific size to make sure the data size matches the printed size
• Do not mix the order of the markers, the order has to match your data description.
• the paper has to be completely flat (tape the paper to a board if need be)
• the more markers the better, but the bigger the better as well.
• make sure to get camera calibration parameters with a board that matches the above description as well.
• it doesnt matter if some of the markers are not detected (the pose works under occlusions), as long as enough are detected (enough is relative :) )
• I printed on an A3 paper. If you have access to an A3 printer I can share my pdfs, but you should be able to create them using the provided code.

[assia855]

Hi @cviviers Thanks for all this suggestsion.

• I measure manually the size of my marker and I found 6cm, so I change the parameters in detect_Charuco_pose_board to number_markers_x = 4, number_markers_y = 3, square_width = 0.07, aruco_width = 0.06
• For the order of markers where can I check it please and how can I fix it
• I make my paper completely flat and that helps me a lot in detecting more aruco and I have the pose for more frames than before now
• I get the camera calibration parameters with calibrate.ipynb and I think the pose is better than before but it still far from the real pose
• I tried as well to play with rotation_offset and translation_offset but I still have this kind of pose

[cviviers]

You need to print the exact pdf/png that is produced by the script. Do not cut the markers out and put them in different places or on a new piece of paper. The distance between markers also matters. Your latest pictures also seem like the paper is not flat enough.

I think the offset you are referring to is the offset to the object from marker 1. Change this when you see the correct axis of the board in the corner of the board being displayed.

[assia855]

Hi @cviviers thanks a lot for your replay, I printed the charuco board and recompute the camera parameters with calibrate.ipynb this is the parameters that I got: {"distortion": [-0.04745589091727264, 0.0004570602824454296, 0.012062003741561756, 0.008462426231645262, -1.1409376245420838e-06], "intrinsic": [[133.8815378174932, 0.0, 326.066571194312], [0.0, 81.07665840500025, 239.59919600396822], [0.0, 0.0, 1.0]]} and after running create_dataset.ipynb with your parameters this is the projection that I got:

What can I fix as well to have the exact pose please?

Thanks a lot for your help.

[cviviers]

what aruco dictionary did you use? remember since you created a custom board, you should use that custom board in all the other scripts as well.

so for calibrate: instead of the current (# aruco_dict = Aruco.getPredefinedDictionary(Aruco.DICT_6X6_50) board = Aruco.CharucoBoard((8, 5), 0.5, 0.3, aruco_dict) ), use the board description you printed. the same holds for the create dataset.

I can already see you are only using Aruco.DICT_6X6_50, instead of the Aruco.DICT_6X6_250 used in the create_board script.

[assia855]

Hi @cviviers thanks for the remark, yes actually you are right I didn't pay attention to that but actually in your code calibrate for the board you have: board = Aruco.CharucoBoard((8, 5), 0.5, 0.3, aruco_dict) and in pose_utils you have: def detect_Charuco_pose_board(img, matrix_coefficients, distortion_coefficients, markerSize = 6, totalMarkers = 50, number_markers_x = 8, number_markers_y = 5, square_width = 0.05, aruco_width = 0.03)what's the reason for this mismatch between the two scripts for aruco_width and square_width parameters.

1. I change in calibrate script :

   aruco_dict = Aruco.getPredefinedDictionary(Aruco.DICT_6X6_250) 
   board = Aruco.CharucoBoard((10,10), 0.03, 0.02, aruco_dict)

Then the new camera parameters are:

• {"distortion": [-0.14586251322343352, 0.002714409119961078, -0.03316363613690248, -0.0301155824088724, 1.8753035462026414e-05], "intrinsic": [[218.72184535984664, 0.0, 375.74087853024554], [0.0, 312.4675978456318, 417.9311421430765], [0.0, 0.0, 1.0]]}

2. I change in pose_utils the parameters in function detect_Charuco_pose_board:

def detect_Charuco_pose_board(img, matrix_coefficients, distortion_coefficients, markerSize = 6, totalMarkers = 250, 
                            number_markers_x = 10, number_markers_y = 10, square_width = 0.03, aruco_width = 0.02)

but I still have this kind of projection:

What I'm still doing wrong please? Thanks a lot :)

[cviviers]

You do not have to use the same board to calibrate your camera and to eventually acquire data with. You can just use a standard checkerboard to calibrate the camera. In your implementation above, do you also use theboardpts, num from when you created the custom board in the follow up code?

I think to ensure everything is well calibrated and to acquire the board pose correctly, lets start with the basic charuco board first (no cup included). From now on, use the board you describe here

aruco_dict = Aruco.getPredefinedDictionary(Aruco.DICT_6X6_250) 
board = Aruco.CharucoBoard((10,10), 0.03, 0.02, aruco_dict) 

in all your scripts and also reprint it. It will have no hole in the middle but thats okay.

This should give you the correct poses I believe.

[cviviers]

I will try to update the getting started code in the coming weeks to streamline the process. I think for people that are new to aruco markers and pose estimation there are quire some pitfalls. Your feedback helps a lot :)

[assia855]

Hi @cviviers thanks for your answer, yes I use as well the boardpts, num to generate the charuco board.

I follow your suggestion:

1. First I generate a charuco board without hole and use calibrate.ipynb to generate camera_parameters.json file
2. Second I put my cup in the middle and estimate therotation and translation offset than I continue the process
3. Third I remove ' /1000 ' from verticesog = np.c[np.array(mesh.vertices)/1000, np.ones((len(mesh.vertices), 1))].transpose() # vertices in object coordinate in meters in the part of load 3D model

for the projection I got :

but for the mask I got:

Do the projection results indicate that my labeling are correct now? Could you please advise me on how to fix the mask issue?

Thanks for all your help.

[cviviers]

Very good, nice job getting the pose of the board. So the next part becomes a bit more tricky, the current code gives you this box and mask with which you can already train. However, it will likely not work well when deployed in a real world scenario. Do you have an accurate 3D model of the mug? If so, there are a few ways you can project that model onto the image to get an accurate mask

[assia855]

Hi @cviviers thanks a lot. Actually I launch train with this mask it trains and I have a detection on test image but I don't understand why I have can't open/read file: check file path/integrity in terminal while training. Am I missing somthing? should I register the images somewhere first before training?

For the 3D model of the mug I got it from YCB-V dataset. I'm trying to find some codes to project the 3D on my 2D images but I'm not sure it is the good way to that or I should generate a 3D model of my real mug. I took the 3D mug of YCB-V dataset because it was similar to my mug, clean and respect all the details that the neural needs.

thanks so much for your help and illumination.

[cviviers]

This is a discrepancy between the windows and linux pathing convention I think. Somewhere either a \ is being used when it should be a / Delete the cache files and check which file is not using the linux file pathing convention.

[assia855]

thanks @cviviers, I solve this issue effectively it was just issue of / when I switched it the warning disappear.

[cviviers]

I recently thought of a nice method for you to get accurate enough masks if you do not have a good 3D model. You can just use the segment anything model https://github.com/facebookresearch/segment-anything to set up a semi automatic segmentation of your objects. You can just prompt is with "cup" in each image and then check if it gives you good results.

[assia855]

Thanks @cviviers, actually for the mask I used track anything and I generate the masks then I launch a training with it, here's the results.

but I don't know if the rotation and the translation of the object is correct or not since I don't visualize as well the mask on the object?

[cviviers]

If you run the test.py instead of detect.py it will draw the ground truth points as well if you have them

[assia855]

Hi @cviviers I run test.py and it create an empty folder in test I don't understand why ?

Thanks a lot for your help!

[cviviers]

You have to pass the flag --test-plotting when running test.py. Please check the code

[assia855]

thanks @cviviers for your remark. Actually now I have the images with two boxes blue and green but no mesh. Why I don't have the projection of the mesh ?

[cviviers]

The current code does not do that, since we were interested in obtaining the pose using the keypoints. Feel free to extend the code to your needs :)

With this I think you've learnt a lot and completed what your current Github issue set out to do. Congrats! Feel free to open a new issue if you have any more problems with the current code.