Error in evaluation residual block

[cpicanco]

This error occurred when offline calibration started the mapping process. I am guessing it is related to a bad world.intrinsics file.

circle_detector - [INFO] video_capture.file_backend: End of videofile 24133 24134
Reflection detected
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1103 17:45:15.895808  3942 residual_block.cc:131] 

Error in evaluating the ResidualBlock.

There are two possible reasons. Either the CostFunction did not evaluate and fill all    
residual and jacobians that were requested or there was a non-finite value (nan/infinite)
generated during the or jacobian computation. 

Residual Block size: 3 parameter blocks x 3 residuals

For each parameter block, the value of the parameters are printed in the first column   
and the value of the jacobian under the corresponding residual. If a ParameterBlock was 
held constant then the corresponding jacobian is printed as 'Not Computed'. If an entry 
of the Jacobian/residual array was requested but was not written to by user code, it is 
indicated by 'Uninitialized'. This is an error. Residuals or Jacobian values evaluating 
to Inf or NaN is also an error.  

Residuals:             -nan         -nan         -nan 

Parameter Block 0, size: 3

          -0 | Not Computed  Not Computed  Not Computed  
          -0 | Not Computed  Not Computed  Not Computed  
          -0 | Not Computed  Not Computed  Not Computed  

Parameter Block 1, size: 3

           0 | Not Computed  Not Computed  Not Computed  
           0 | Not Computed  Not Computed  Not Computed  
           0 | Not Computed  Not Computed  Not Computed  

Parameter Block 2, size: 3

           0 | Not Computed  Not Computed  Not Computed  
          -0 | Not Computed  Not Computed  Not Computed  
           0 | Not Computed  Not Computed  Not Computed  

Ceres Solver Report: Iterations: -2, Initial cost: -1.000000e+00, Final cost: -1.000000e+00, Termination: FAILURE
Termination Error: FAILURE

[cpicanco]

Now I am guessing the problem is related to using Calibration Mode: 3d for a monocular recording.

[papr]

@cpicanco Yes, this is definitively related to Calibration Mode: 3d since we do not use the ceres solver for 2d but polynomial regression. I am not exactly sure how this error is triggered. It might be due to the solver simply not converging. Are you able to reproduce it?

Also, this caught my eye:

Iterations: -2

[cpicanco]

@papr , can you clarify what does "Calibration Mode" actually means? Is the 3d option for HMD only, or it may apply for binocular in general?

[papr]

The 2d calibration mode uses polynomial regression to find a mapping function that maps pupil norm positions to gaze positions. The 3d calibration mode uses the bundle adjustment method to find the physical relation between the eye cameras and the world camera. Afterwards the 3d vector gaze mapper transforms the pupil normals from the eye coordinate system to the world coordinate system.

This is available for Pupil Headsets for some quite some time now. The 3d HMD calibration makes use of the same procedure but is still in active development.

Edit: Ceres is used to solve the bundle adjustment problem.

[cpicanco]

So, for binocular in general. I increased the memory size from 2gb to 4gb. Now I can open and run offline calibration and offline pupil detection faster. I am trying to reproduce the error with no success until now. This one is from a monocular recording (I am not sure how to interpret the "3d calibration mode" for monocular recordings, but here it is):

Ceres Solver Report: Iterations: 8, Initial cost: 4.390676e+00, Final cost: 7.879207e-01, Termination: CONVERGENCE

[papr]

3d calibration works in the same way for binocular and monocular data. Binocular data has the advantage that you can calculate depth using vergence.

[cpicanco]

Ok, thanks! What about "Calibration Mode:" -> "Polynomial regression" | "Bundle adjustment"?

[cpicanco]

I could reproduce the error only with one of my recordings. I will close this issue, reopen at will.

[fneitzel]

Thank you. This is really helpful to understand.

So can I also calculate depth with 2d binocular? Do i need to stick to a plane area for binocular calibration for 2d/3d?

[cpicanco]

@fneitzel , yes you can calculate depth with 2d binocular.

@papr , I noticed that "Calibration mode: 3d" will be auto changed to "2d" if Offline Pupil detector was done with 2d method. Is this by design?

[papr]

3d vs 2d

The 3d mapper requires 3d pupil data. The mapper transforms the pupil vector to a gaze vector. This is not possible by using 2d normalized positions. Therefore, yes @cpicanco, it is by design that the calibration mode is set to 2d/polynomial regression if the pupil detector is set to 2d.

monocular vs binocular

@fneitzel @cpicanco We cannot calculate depth using the 2d mapper. Neither monocular nor binocular. The 2d mapper only yields 2d normalized positions. (There is a special case for the hmd add-ons, but this is a different discussion.)

In the 3d binocular case we can calculate depth using vergence using the two mapped gaze vectors.

Calibration plane

2d case

The polynomial regression case assumes the markers to be on a plane and the resulting 2d mapper will yield 2d locations that are valid on the same plane only. There is some leeway here. The plane does not need to be perfect nor does the subject need to look at objects in this plane. But violating these assumptions will lead to less accurate results.

3d case

The bundle adjustment can calibrate using markers with different depth. The markers do not have to be on a plane. We undistort the marker reference positions using the camera calibration since 196adc4794ff545ad1e3b91caf3d323c7c989ec7 and are able to calculate 3d marker locations. These locations do not correspond to actual 3d location of the marker but lie on planewith distance 1 to the camera frustum in the world camera coordinate system. @marc-tonsen Please correct me if I am incorrect.

[cpicanco]

Thanks for the clarification.