3D reconstruction - 2nd camera distortions ignored

[Witek-]

Issue Summary

The precompiled OpenPoseDemo v1.6.0 and v1.7.0 seems to ignore second camera distortion parameters. No matter what I put as distortions, there is no difference in the displayed image. If I set exaggerated distortions to the first camera - both views are heavily and identically distorted, like so: If the distortion parameters for the first camera are correct and no matter what I put for the second one, I get:

Executed Command (if any)

bin\openposedemo --video lr.mp4 --3d_views 2 --3d --frame_undistort true --number_people_max 1 --output_resolution 640x512

Type of Issue

Help wanted

System Configuration

Operationg system: Windows 10 Graphics card: Nvidia GTX1060 6GB Driver Version: 461.40 from 26.01.2021 CUDA 10.1, CuDNN: 8.0.5 Visual Studio Community 2017 Caffe, OpenCV: Default from OpenPose OpenPose prebuit demot 1.6 and 1.7

Here are my camera parameters:

Camera 1:

<?xml version="1.0"?>
<!-- Note: Camera matrix translation units are meters. -->
<opencv_storage>
<CameraMatrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>4</cols>
  <dt>d</dt>
  <data>
     1. 0. 0. 0. 0. 1. 0. 0. 0. 0. 1. 0.</data></CameraMatrix>
<Intrinsics type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    1534.4799 0. 628.82324
    0. 1533.6852 531.02933
    0. 0. 1.</data></Intrinsics>
<Distortion type_id="opencv-matrix">
  <rows>8</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -0.23823735 0.18993686 0. 0. 0. 0. 0. 0.</data></Distortion>
</opencv_storage>

Camera 2:

<?xml version="1.0"?>
<!-- Note: Camera matrix translation units are meters. -->
<opencv_storage>
<CameraMatrix type_id="opencv-matrix">
  <rows>3</rows>
  <cols>4</cols>
  <dt>d</dt>
  <data>
   0.99995741 0.0019467166 -0.009021109 89.022671e-03 -0.0018696433 0.99996175 0.0085442351 0.40566657e-03 0.0090373971 -0.008527005 0.9999228  -1.0467076e-03</data></CameraMatrix>
<Intrinsics type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>d</dt>
  <data>
    1536.2698 0. 629.11479
    0. 1534.9349 528.87344
    0. 0. 1.</data></Intrinsics>
<Distortion type_id="opencv-matrix">
  <rows>8</rows>
  <cols>1</cols>
  <dt>d</dt>
  <data>
    -0.24999544 0.25860068 0. 0. 0. 0. 0. 0.</data></Distortion>
</opencv_storage>

No matter what I put as distortion for the second camera, the result is the same. The 3D reconstruction is also incorrect. What am I doing wrong?

[gineshidalgo99]

Very probably it's a bug in the implementation that has always being there, and which people might have missed because using similar cameras usually result in very similar distortion parameters...

I do not have any 3D videos at the moment, could you send me the video and camera parameters? You can email them to me at gines@alumni.cmu.edu if you do not wanna share them publicly, and I will take a look into fixing it. Sorry about the issues!

[Witek-]

I did send you an email 4 days ago. Hope you received it.

[GirardMatthew23]

How did you even get the extrinsic calibration tool to compile? I've tried everything, and am getting massive errors on both Linux and Windows. Documentation on extrinsic calibration tool is light, and the portable doesn't contain an extrinsic calibration utility.

[Witek-]

Well, I used cmake-gui on Windows and it did compile without a problem (I built projects separately, Only project->compile). So I am able to compile it, but the calibration results are incorrect as described here: https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1874