Calibration Rig

[botaohu]

• [x] a rig for (lens) optical calibration to be designed by outer sourcing supplier. Waiting for quotation from supplier. ETC: TBC.
• [x] CNC it and ship it to Amber's place.
• [ ] generate the calibration file and pre-distortion chart by Amber
• [ ] heyhey!

[botaohu]

@nicolehong0 initial meeting Mar 14, 2020 3pm.

[nicolehong0]

meeting note for calibration rig discussion on 2020/3/14. optical calibration rig discussion_20200314.pptx

[botaohu]

ChArUco chart: https://calib.io/pages/camera-calibration-pattern-generator

We will use Zed 2 camera https://support.stereolabs.com/hc/article_attachments/360056836934/ZED2.step

[botaohu]

@nicolehong0 can you upload the apple's slides here?

[nicolehong0]

@botaohu this is the A2 OQC test intro. Lens OQC 10.2.19.pdf

[nicolehong0]

A2 optical variability study fixture used as below:

[botaohu]

Product Requirement Document for AR HME Calibration Rig

Definition

• Coordinate system

  • World coordinate system ${G}$.
  • Phone coordinate system ${P}$.
  • ARHME coordinate system ${H}$.
  • Test camera coordinate system ${C}$.
  • Calibration object coordinate system ${O}$.
  • Calibration rig coordination system ${R}$.
    • Axis system of ${R}$ following right-hand rules:
      • $z$ axis aligns with the direction from camera to calibration object.
      • $y$ axis is against with earth gravity.
      • $x$ axis points to leftward.

• Key Position

  • Ideal eye position $^{H}\mathbf{p}_{Ei}$
  • Actual eye position $^{H}\mathbf{p}_{Ea}$
  • Test camera optical center $^{C}\mathbf{p}_E$ for matching eye position

• Symbols

  • $^{G}\mathbf{p}_{I}$ denotes a position of ${I}$ in ${G}$ coordinate system.
  • $^{G}_{I}\mathbf{q}$ denotes a rotation quaternion from ${I}$ coordinate system to ${G}$ coordinate system.

Objective

Design and make a calibration rig to place phone, ARHME, test camera, calibration object in the system in order to match the virtual rendered images with the physical calibration object from test camera's perspective.

There are two major testing phases in the calibration.

• Static Phase: when the whole system is stationary.

  • The purpose for this phase is to test the rendering software in the phone so that we need to make sure that we can render the image correctly to match the calibration object.

• Dynamic Phase: when ARHME + phone + test camera is moving together while calibration object is stationary.

  • The purpose for this phase is to test the tracking software in the phone so that we need to make sure that we can tracking the position of the phone in the physical space correctly. The image will be rendered correctly to match the calibration object while moving the ARHME.

Assumptions:

Known information

• ideal 3D geometry of ARHME, phone, test camera, calibration object.
• 3D geometry of ideal light tracing from ARHME optical system includes
  • ideal eye position
  • viewport
  • distortion
• camera transformation of test camera. See https://www.mathworks.com/help/vision/ug/camera-calibration.html
  • instrintic parameters
  • extrinsic parameters
  • distortion parameters

Phone static rendering:

Necessary input information for static rendering:

• the position $^{H}\mathbf{p}{O}$ and rotation $^{H}{O}\mathbf{q}$ of calibration object in ARHME coordinate system ${H}$.
• user's interpupil distance: the offset of $x$ axis of $^{H}\mathbf{p}{Ea} - ^{H}\mathbf{p}{Ei}$. $x$ distance from the ideal eye position to actual eye position.

Given above information, renderer can render the virtual image of calibration object based on the ideal optical system of ARHME.

Phone dynamic rendering:

Additional to static rendering, the software will estimate the pose (position + rotation) of the phone $^{G}\hat\mathbf{p}_{P}$ to renderer to render a static images 60 frame per second.

The software needs to predict the predict pose before rendering start to reduce the latency to match the virtual image to the actual object.

For example, 60 frame per second rendering refresh rate needs 16.6ms rendering time per frame. $\Delta t = 1000ms / 60$

If we want to display an image at time $t$, we need to predict the pose $^{G}\hat\mathbf{p}_{P}(t)$ at time $t0 = t - \Delta t$. The goal for dynamic rendering is to minimize the error $^{G}\hat\mathbf{p}{P}(t) - ^{G}\mathbf{p}_{P}(t)$ in order to match virtual and real object.

Design

Calibration Rig Design

• Calibration rig can hold the ARHME as precisely as possible while with the smallest pressure and stress on the ARHME.
• Calibration rig has physical stopper to align the phone in $x$ axis.
• Calibration rig should be designed to easily swap ARHME. Should be < 1minute work to switch ARHME.
• Calibration rig should have mark to align with Test camera rig to indicate the ideal eye center.
• On the side of calibration rig, we should have a horizontal mark at the eye position on $y$ axis to level the whole system including test camera, and calibration rig.

Test camera

• Test camera should be global shutter camera for dynamic phase test.
• The sampling rate of test camera should be equal or higher than 60fps.
• Test camera should be a color camera to calibrate the chroma-abberivation.
• Test camera's form factor should be small enough to fit in two camera side by side for 58 to 72mm IPD. i.e. Test camera's $x$ axis width should be <= 29mm.
• Test camera resolution should be high enough to distinguish 0.5 degree difference at 0.75m object.
• Test camera's lenes should be low-distortion lens with the full coverage FOV of ARHME which is 48 x 40 degree. Recommended focal length: 6mm.
• We could adjust the focal distance of lens from 0.5m to 5m.

Test camera rig:

• Test camera rig should adjust in $x \pm 10mm$, $y \pm 10mm$, $z \pm 10mm$ axis for moving the test camera within the eyebox of the ideal eye position.
• Test camera rig should easily align with the ideal eye position.
• Test camera rig should have marks that we can easily read the $\Delta x, \Delta y, \Delta z$ of the offset from the ideal .

Calibration object

• The size of calibration object should cover the ARHME FOV at the focal plane.
• Calibration object should a plane at the beginning.
• For dynamic phase, the calibration object could a object.
• We could easily read the relative position of the base of calibration object to ARHME coordinate $^{H}\mathbf{p}_{O}$.
• Calibration object has a center marker to align with the optical axis of the ARHME.

Testing Phone.

This calibration should be designed to be easily compatible with all the phone with different size.

• Preferred model:
  • iPhone 11 Pro.
• Compatible models:
  • All iPhone X or above: iPhone X, iPhone XS, iPhone XS max, iPhone 11, iPhone 11 Pro, iPhone 11 Pro max.

Testing Process.

• Basic steps.

  1. We put ARHME in the calibration rig.
  2. We put the phone in ARHME and align the edge with the stopper.
  3. We put the test camera on the test camera rig. On the test camera rig, we read out the offset of the test camera optical center from the ideal eye position $(\Delta x, \Delta y, \Delta z) = ^{H}\mathbf{p}{Ea} - ^{H}\mathbf{p}{Ei}$.
  4. We input the phone with the ipd with $\Delta x$.
  5. We put the calibration object at the ideal focal distance. The base point of calibration object should be aligned the optical main axis with laser aligner.
  6. We input the phone with the pose of the calibration object relative to ARHME $^{H}\mathbf{p}{O}$ and $^{H}{O}\mathbf{q}$.
  7. The phone renderers the image.
  8. Test camera read out the combination of virtual image and actual image.
  9. The calibration software can compute the difference between virtual image and actual image. We expect the correction calibration can reduce the error under 0.5 degree at 0.75m.

• Static phase.

  1. We only capture the frame staticly. And then we calculates the error between virtual image and actual image

• Dynamic phase.

  1. We will move the calibration rig + test camera rig with ARHME and test camera by hand or robot arm.
  2. We will capture the test camera at 60 FPS. And then we capture the frame for a period. And then we calculates the error between virtual image and actual image as a sequence.
  3. The movement should be tested in the each of six degree of freedom 6DOF.

rig.pdf

[botaohu]

VR Rig for our lens and calibration.

VR jig.step.zip

[botaohu]

@nicolehong0 please make an initial verison.

[botaohu]

@nicolehong0 whats' the ETC?

[nicolehong0]

• 4/10/2020, released CAD for HME calibration system parts (4 parts) 3D printing, CAD in folder HME_calibration_rig_3D_printing_041020.
• 4/10/2020, released HME calibration system CAD for internal review and reference, inspection_system_041120.
• 4/11/2020, released HME calibration system assembly introduction, Calibration_rig_system_assemble_041120.
• 4/17/2020, released CAD for updated HME calibration system parts (3 parts) 3D printing, CAD in folder HME_calibration_rig_3D_printing_041720.
• 4/21/2020, release HME & Fresnel Lens calibration system CAD for internal review and reference, Calibration_System_042120.

Released CAD: https://github.com/holokit/HoloKitX-Optics/tree/master/Internal_Optical_Calibration_System/HME_Calibration_System/Release

[nicolehong0]

Set iPhone 11 Pro rear camera 1 keepout cone tip as the origin of coordinates. The location of real chart and designed eye as: iPhone 11Pro main camera coordinates.pptx

[nicolehong0]

Some key dimension as:

[nicolehong0]

The CAD and introduction files for the calibration system (HME+real chart+device) with the coordinate system which use main camera as the origin for different device (such as iPhone 11 Pro, iPhone 11 Pro Max,, etc) can be found by link: https://github.com/holokit/HoloKitX-Optics/tree/master/Internal_Optical_Calibration_System/HME_Calibration_System/Release/HME_device_realimage

[botaohu]

Close due to new optical calibration system required.