calibrate camera

[Armandpl]

• it looks like blender doesn't simulate lens distortion. there is a node to do so but I don't know if there is a unit to it. it would probably be better to correct the actual images coming from the camera
  • check how to do this
  • check how long it takes to do it on the jetson nano
• for now I roughly measured the translation and rotation of the camera relative to the car. could use PnP to measure it more precisely. build a rig to secure the car, put points on the floor
• the output of the two steps above should be committed to github and we could then pull them e.g from the blender script or the inference script (to correct distortion)
  • we might want to track it w/ artifacts? e.g to know which one was used by blender and therefore what params we should use to project the points when vizualizing preds
  • or just hardcode it, works as well

[Armandpl]

I think the reason why the sim and actual cam look different is the lens distortion and maybe difference in extrinsinc parameters. put the sim camera and real car on the track at roughly the same position and compare the images

[Armandpl]

actually I could also measure lens distortion and then distort the sim image lmao maybe that's better, it removes one step in the inference pipeline

[Armandpl]

i dont understand why we estimate K when calibrating? isn't it just defined by the focal length when skew is null? is it only useful when there is skew? or do camera have small variation in focal length? do the calibration and see what's in the estimated K

[Armandpl]

understand how jetcam works, how are images resized, how does the fact that I can set any framerate works? can the cam do any framerate?

[Armandpl]

digging

• when juxtaposing an image taken in 640x480 on top of a (resized) 1280x720 image they do match.
  • sensor is 3.691 mm x 2.813 mm which is ~= 640/480 so I'm wondering if the 1280x720 images are just resized?
• looking into jetcam code looks like we're capturing 640x480 images then resizing to 224x244
  • also looks like they're using bgr, unsure if I am/was swapping channels before feeding the image to my pytorch model doesn't look like it lmao

[Armandpl]

quick calib gave this matrix for the camera from 640x480 images:

array([[269.24624039,   0.        , 345.9362606 ],
       [  0.        , 357.38497645, 256.33152125],
       [  0.        ,   0.        ,   1.        ]])

F(mm) = F(pixels) SensorWidth(mm) / ImageWidth (pixel) F_x (mm) = 269 3.691 / 640 = 1.55 mm F_y (mm) = 357 * 2.813 / 480 = 2.09 mm

doesn't seem to match 0.87 from the datasheet? Ok so it's because the calibration process can't guess the scale of things so the units don't matter. We can specify the sides of the squares if we want translation and rotation vectors to mean anything.

[Armandpl]

Ok so I'm starting to have a plan to match sim images and real images:

• measure cam position with pnp
  • place chessboard taped to book against front wheels so there is no rotation between the car and the chessboard
  • do pnp, this will give use camera tilt, pan and roll
  • z (top) translation (don't forget to add book thickness)
  • and maybe y (right) if the book is somewhat centered?
  • won't really give us x (forward) translation if we don't measure the distance between the book and center of mass. I think we can measure x translation by hand for now and we'll see if we want to improve this later.
  • basically we'll need some sort of rig to know and fix the x, y translation and rotation wrt the chessboard.
• measure distort coefs with cv2 calibration
• blender: use focal len, sensor size from datasheet
• add lens distortion with kornia, paste the (distorted) car hood as a mask and lets go
  • generate blender images in 320x240 to save disk space so we'll need to scale dist coefs by 1/2 (i think)

[Armandpl]

According to v4l2-ctl --list-formats these are the available modes for my imx219 camera:

ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'RG10'
        Name        : 10-bit Bayer RGRG/GBGB
                Size: Discrete 3264x2464
                        Interval: Discrete 0.048s (21.000 fps)
                Size: Discrete 3264x1848
                        Interval: Discrete 0.036s (28.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1640x1232
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)

gst_argus on the other hand lists the following modes:

GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

I am tempted to use 1280x720 because it is the smallest resolution and I'm going to resize to 224x224 so might as well save some bandwith, however:

• I won't need more than 60 fps and probably I'm going to start with 30 fps so unsure how this would work with jetcam and nvargus. Is is choosing the closest res/fps and sticking with it? is nvenv reencoding the stream with a different framerate?
  • measure framerate coming out of jetcam
• 1280x720 is cropped but I'm not sure how, furthermore it's not cropped like 1920x1080 and I need at least 1920x1080 to calibrate the camera

[Armandpl]

alright alright I got it:

• [x] find the crop factor between full sensor and 720p. overlay 720p image on top of 3264 image and find the size (w, h) of the center crop by resizing the 720p image
  • 2560x1440
• [ ] get calib chessboard and pnp chessboard in 3264x2464
  • crop using the values found above
  • calibrate
  • pnp and update camera offset and rotation
• [ ] in blender get full sensor image at the smallest possible res before 224x224
  • then add crop node to crop like the 720p image
• [ ] find how to rescale mtx and dist matrices from crop size to 720 and from crop size to sim image size
  • once that's done warp the sim images
  • and add hood mask on top
  • make the warping masking torchvision transforms
• [ ] place car on track in real life and in sim and compare
• [ ] gen traj, add augmentations and train
  • maybe optim the fact that if we want e.g 15 steps trajectories it means the last 15 images of each traj are useless
  • if we have traj of like 45 images it means 1/3 of the rendered images are useless it's kind of shit
  • but its a trivial problem, feed it to gpt4
  • use distortions coefficient to properly project trajectories on images

[Armandpl]

ok so the values given by v4l seem to be the oned we should look at, I can't actually start the cam at e.g 120 fps. but the modes specified by v4l all seem to work the nvargus modes also don't match what's actually running? e.g if I ask for 720p 60 fps it is going to say 720p 120fps but then only actually provides 60fps

it also look like cv2 or gstreamer is able to change the framerate from 60 to whatever

[Armandpl]

ok so turns out there is a fisheye lens in cycles in blender, can use that instead of distorting the image after generating?

• but unsure what the units are?
• are shift x and shift y tangential coefs? if not where are they? -> don't think they are
• opencv has a fisheye model, do I need to select it when calibrating? I think so -> look at this tutorial or this gist to do it
• maybe I can simulate the radial distortion in blender and add the tangential one with kornia? this way it's faster to generate bc I can use eevee instead of cycles?

Action plan:

• [x] calibrate using the fisheye model
• [ ] try out polynomial lens
  • if it doesn't work, understand how the lens work in eevee? does it make sens to distort it after the fact? or is it already somewhat distorted?

[Armandpl]

update update:

• looks like I can distort images using opencv fisheye model but not sure if this maps correctly to the real images?
• try vp calib to be able to check if images match
  • also viz on all sim/real images
  • maybe list poses so I can move the cam

[Armandpl]

right right looks like i can distort the sim images w/ the fisheye calibration coeffs! and it matches the sim only issue now is the distorted image doesn't span the whole real image. why is that the case?

[Armandpl]

contexte et problèmes pour poser des questions à des gens:

• Objectif: matcher une caméra Blender avec une caméra réelle pour générer un dataset synthétique et entrainer un réseau de neurones. à priori on préfère matcher la caméra simulée à la réelle plutot que la réelle à la simulée (=calibration+undistort) car on a peu de ressources sur le système embarqué (jetson nano)
  • 
  • cela dit on a pas benchmark si ça prendrait du temps de calibrer sur le syteme embarqué. notre nn tourne a ~60fps donc il faudrait pouvoir undistort à la même fréquence
• approche actuelle:
  • trouver les paramètres intrinsèques de la cam K et les coeff de distortion D de l'objectif avec le module de calibration fisheye d'openCV (plus d'infos sur leur modèle de fisheye)
  • trouver les paramètres extrinsèques (position et orientation de la cam) par mesure avec un pied à coulisse, la technique du vanishing point et avec trial and error pour que les images réelles et simulées se superpose bien
  • matcher la caméra blender avec la datasheet du sensor: distance focale et dimensions du sensor (+ shift x, y trouvés à la main par itération et comparaison avec les images réelles)
  • générer une image avec blender en utilisant le pinhole model donc aucune distortion
  • 
  • distordre l'image selon les paramètres trouvés lors de la calibration
  • 
• problèmes de l'approche actuelle
  • de ce que je comprends, du fait de la façon dont les rayons arrivent à un objectif fisheye on peut pas totalement le simuler une fois qu'on a généré une image avec le pinhole model et donc mon image final recouvre pas toute l'image réelle. je vois trois façon d'aller de l'avant
  • 1. cropper dans cette image et accepter de perdre une partie des données recoltés par la cam. potentiellement handicapant en pratique car on pourrait rater des virage j'imagine? potentiellement ça passe mais en tout cas peu souhaitable
       • cropper une video sur la piste pour se rendre compte si on perds beaucoup d'infos ou pas. eventuellement crop ça résout aussi le pb d'avoir le capot sur l'image
  • 2. simuler le fisheye dans blender -> render beaucoup plus long (il faut qu'on bench si c'est meme possible avec la RTX de Rian, envisageable de louer une machine sur une render farm), pas encore réussi à matcher la cam
       • scripter de changer les parametres du fisheye avec bpy, prendre images simulées d'un chessboard, calibrer avec openCV, comparer K et D des images simulées avec K et D réels jusqu'à ce que ça match. ou juste faire ça a la main
  • 3. calibrer sur le systeme embarqué
• Questions:
  • quelle approche semble la plus prometteuse parmis les trois au dessus? if any?
  • est-ce qu'il y a des tricks connus pour aider à la généralisation avec les datasets synthétiques? e.g randomiser les paramètres de la caméra

[Armandpl]

try cropping, training network, deploying

• [x] work out crop size and location
  • crop in the whole image 3264x2464
  • w, h = 1500x1000 px
  • location: x=1748, y=1175
  • crop in the 2560x1440 region
  • w, h = 1500x1000
  • x = 1397, y = 663
  • crop in the 1280x720 image
  • w, h = 750x500
  • x = 698, y = 331
  • left, top = 314, 81
• [x] crop onboard video and see if it's enough
  • it's meh but lets try
• [ ] gen 250k traj + images
• [ ] work out how not to waste images
• [x] work out generation pipeline
  • smallest render size 720 p crop -> 640x360 this makes the end crop 375x250
  • this makes the full size render image 816x616
  • and the crop in the rendered image:
    • w, h = 375x250 px
    • location: x=437, y=294
  • generate full sensor size (but at smaller res), warp then crop
• [ ] project ground truth/pred on cropped images
• [ ] sample traj and render eevee
  • work out smallest render size
  • try warping
  • bench eevee 250k