Camera Calibration Error: IndexError: index 13 is out of bounds for axis 0 with size 13... Error with code? or Process?

[f5ces27er5c767s]

Hi there @linchangyi1 ,

Thank you so kindly for providing the code! What a wonderful project :)

We managed to get the sensor hardware working and are now at the camera calibration step. Getting the raw image saved was okay but in the next step where I was asked to push the calibration board on the sensor and then press "y" I get this bug. Do you happen to know how to resolve this?

This was my terminal line and error:

Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
yyTraceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 13 is out of bounds for axis 0 with size 13

Is there an error in the code? Am I supposed to push down really hard or do I only need to do it slightly? Do I need to just push until I see all dots of the calibration board on the sensor?

Thank you kindly @linchangyi1 , please let me know if you resolve this! I also attach an image of my process.

[linchangyi1]

Hi @f5ces27er5c767s , I think you probably need to push it down really hard for clear imprints. Btw, have you attached the image? I didn't see that

[f5ces27er5c767s]

Hi @linchangyi1 Thank you for replying back to me so quickly, so sorry for the late reply it got busy here haha.

The image of the calibration board for this camera calibration step is as follows:

I tried your approach of pushing it down really hard and I get the same errors, now though with differing IndexError (i.e. the number of index out of bounds is different each time). For 2-3 really hard presses I get the following output:

(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py 
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 46 is out of bounds for axis 0 with size 46
(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py 
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 50 is out of bounds for axis 0 with size 50
(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py 
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 48 is out of bounds for axis 0 with size 48
(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ 

Any suggestions? I am pressing it so hard that the calibration board now has made marks on the top gel. I don't think it's my sensor, do you happen to know how we can resolve this? :)

Thank you! @linchangyi1

[linchangyi1]

Hi @f5ces27er5c767s ,

Actually you tactile image for calibration is not clear enough although you have pressed the board very hard. I think this is because you haven't adjust the camera focus very well. As for a good calibration image, refer to the tactile image for calibration in the paper.

The marks on the top gel should disappear soon after you remove your calibration board. If you are facing that problem, you may need to check whether you choose the same material as that used for the original 9DTact.

[f5ces27er5c767s]

Hi @linchangyi1 by "adjusting the camera focus" apologies for the simple question, but do you have any recommendations as to how to do this? :)

I am not quite sure how to adjust the camera focus, do you mean I need to open the camera up and fix the lens? Or is there a program/file you know of that could fix this camera focus?

Thank you kindly for your message replies, they are really helping! @linchangyi1 :)

[abtabrizi]

hi @linchangyi1, with regards to objects leaving imprints on the black gel, I faced the same issue. I used same the materials listed in the paper, however, for the black pigment I used Smooth-on Black Slic Pig as opposed to the one you recommended.

The marks on the top gel should disappear soon after you remove your calibration board. If you are facing that problem, you may need to check whether you choose the same material as that used for the original 9DTact.

[linchangyi1]

Hi @linchangyi1 by "adjusting the camera focus" apologies for the simple question, but do you have any recommendations as to how to do this? :)

I am not quite sure how to adjust the camera focus, do you mean I need to open the camera up and fix the lens? Or is there a program/file you know of that could fix this camera focus?

Thank you kindly for your message replies, they are really helping! @linchangyi1 :)

Hi @f5ces27er5c767s, yes you need to rotate the lens of the camera until you see a clear tactile image.

[linchangyi1]

hi @linchangyi1, with regards to objects leaving imprints on the black gel, I faced the same issue. I used same the materials listed in the paper, however, for the black pigment I used Smooth-on Black Slic Pig as opposed to the one you recommended.

The marks on the top gel should disappear soon after you remove your calibration board. If you are facing that problem, you may need to check whether you choose the same material as that used for the original 9DTact.

Hi @abtabrizi, I am not sure if it's caused by the pigment you chose. One point I could recommend is to ensure that the gel is fully "dry" after curing. I also suspect that if your printer has low accuracy or if the calibration board has sharp edges, it could be contributing to the problem.

[abtabrizi]

@linchangyi1 you are correct, I printed the board using PLA on an old printer and the dots on the board were sharp. I will try nylon printing.

I also suspect that if your printer has low accuracy or if the calibration board has sharp edges, it could be contributing to the problem.

[f5ces27er5c767s]

Hi @linchangyi1 thank you for messaging, we adjusted the focus of the camera and fixed it but in running the python Camera Calibration we still get the same issue. I tried it various times now with 1) fixed camera focus and 2) making sure to press hard on the sensor, the outputs I get still result in some IndexError:

(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 58 is out of bounds for axis 0 with size 58
(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 61 is out of bounds for axis 0 with size 61
(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
^CTraceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 33, in run
    sample = self.camera.get_raw_image()
  File "/home/9DTact/shape_reconstruction/camera.py", line 44, in get_raw_image
    return self.cap.read()[1]
KeyboardInterrupt

(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 58 is out of bounds for axis 0 with size 58
(9dtact) Legion-Pro-5-16IRX8:~/9DTact/shape_reconstruction$ python _1_Camera_Calibration.py
------Camera is open--------
DON'T touch the sensor surface!!!!!
Please press "y" to save the reference image!
Reference image saved!
Please press the calibration board on the sensor and press "y" to save the sample image!
Traceback (most recent call last):
  File "_1_Camera_Calibration.py", line 180, in <module>
    cc.run()
  File "_1_Camera_Calibration.py", line 42, in run
    self.calibrate_image(ref, sample)
  File "_1_Camera_Calibration.py", line 92, in calibrate_image
    (all_point[i*self.col_points+j + 1][1], all_point[i*self.col_points+j + 1][0]), command_color,
IndexError: index 52 is out of bounds for axis 0 with size 52

WIth the camera focus fixed, what could it be? Thanks @linchangyi1 for all your help kindly. We appreciate you :)

[f5ces27er5c767s]

@linchangyi1 This is my images with the camera focus fixed:

left is without calibration board, right with calibration board pressed really hard too:

[linchangyi1]

Hi @f5ces27er5c767s,

The total number of point candidates should be 7 * 9 = 63. The issue arises because fewer than 63 points are being detected. One possible reason is that the imprints near the edges of your tactile image aren’t clear enough—please refer to the calibration image in the paper. Additionally, please double-check the camera’s focus and the printed calibration board.

[f5ces27er5c767s]

Hi @linchangyi1 thank you kindly, we managed to finally get the 63 points recognized and detected by being more careful with the gel layer dimensions and also found a darker lower light room helped. We wanted to know if the images/processes we went through below are what we should and you expect :)

Below went through 1. camera calibration, 2. sensor calibration with a 4mm radius ball, 3. shape reconstruction:

Specifically in the end of the camera calibration process it just outputs: [234, 315, 0.055848595538478304]

The questions I want to ask are as follows:

1. what is the next step? We did all calibration steps now. Does finishing the calibration save all these results to some config for the model and then we can just automatically run the force estimation? What are the steps/commands for that?
2. Is the shape reconstruction image okay? Does this just output the shape, which in this case here is just the circular ball?
3. what exactly are the numbers in [234, 315, 0.055848595538478304]? are they saved?

Thank you kindly @linchangyi1 I appreciate your time, efforts and help very much :) You are awesome.

[linchangyi1]

1. The calibration steps are just for shape reconstruction. For force estimation, you probably need to collect more data with your own sensors and use the released checkpoint as a pre-trained model.
2. I think they are ok based on your calibration image. You can get the height map and point clouds, please read the code for more details.
3. It's position_scale and is saved. Please read the code for more details.

[linchangyi1]

@f5ces27er5c767s It would be helpful if you can share how you finally got the clear tactile image. That definitely helps more people to reproduce 9DTact.