Different viewpoint images back projecting problem using undistorted_color_images.

[Mercy2Green]

Hi, thank you for your great work!

Recently I've been using the undistorted_color_images, undistorted_depth_images, and undistorted_camera_parameters to back-project the Matterport 3D dataset.

However, I found that if I project 2 different viewpoints, these point clouds may interfere with each other, as shown below.

dataset matterport
n_images 2358
depth_directory undistorted_depth_images
color_directory undistorted_color_images

intrinsics_matrix 1076.45 0 631.116  0 1077.19 509.202  0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_0.png 03a8325e3b054e3fad7e1e7091f9d283_i0_0.jpg 0.90525 0.275848 0.323155 -2.99825 0.42464 -0.612795 -0.666455 -14.4532 0.0141878 0.740533 -0.67187 1.33124 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_1.png 03a8325e3b054e3fad7e1e7091f9d283_i0_1.jpg 0.820534 -0.381542 -0.425615 -2.98374 -0.571596 -0.547236 -0.6114 -14.4543 0.000362848 0.744955 -0.667115 1.33115 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_2.png 03a8325e3b054e3fad7e1e7091f9d283_i0_2.jpg -0.0846661 -0.653405 -0.752259 -2.97748 -0.996408 0.0548167 0.0645317 -14.4674 -0.000928868 0.755021 -0.6557 1.33093 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_3.png 03a8325e3b054e3fad7e1e7091f9d283_i0_3.jpg -0.90513 -0.267872 -0.330125 -2.98573 -0.424975 0.591297 0.685393 -14.4794 0.0116044 0.760665 -0.649041 1.3308 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_4.png 03a8325e3b054e3fad7e1e7091f9d283_i0_4.jpg -0.820375 0.389515 0.418642 -3.00023 0.571259 0.525712 0.630309 -14.4783 0.0254291 0.756243 -0.653796 1.33089 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_5.png 03a8325e3b054e3fad7e1e7091f9d283_i0_5.jpg 0.0848417 0.661354 0.74526 -3.00649 0.996035 -0.076351 -0.0456356 -14.4651 0.0267202 0.746177 -0.665211 1.33112 0 0 0 1

intrinsics_matrix 1076.01 0 635.509  0 1076.38 511.999  0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_0.png 03a8325e3b054e3fad7e1e7091f9d283_i1_0.jpg 0.902625 -0.00853222 0.430341 -2.99332 0.430149 -0.017951 -0.902578 -14.4636 0.0154264 0.999802 -0.0125343 1.36432 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_1.png 03a8325e3b054e3fad7e1e7091f9d283_i1_1.jpg 0.82401 -0.00475809 -0.566553 -2.98983 -0.56657 -0.00416342 -0.824001 -14.4639 0.00156227 0.999979 -0.00612788 1.3643 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_2.png 03a8325e3b054e3fad7e1e7091f9d283_i1_2.jpg -0.0785585 0.00907021 -0.996867 -2.98832 -0.996908 -0.000537117 0.0785572 -14.467 0.000177485 0.999958 0.009083 1.36425 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_3.png 03a8325e3b054e3fad7e1e7091f9d283_i1_3.jpg -0.902493 0.0191241 -0.430278 -2.99031 -0.430517 -0.0106985 0.902519 -14.4699 0.0126569 0.999759 0.0178871 1.36422 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_4.png 03a8325e3b054e3fad7e1e7091f9d283_i1_4.jpg -0.823839 0.0153494 0.566614 -2.99379 0.5662 -0.0244859 0.823903 -14.4697 0.0265208 0.999582 0.0114802 1.36424 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_5.png 03a8325e3b054e3fad7e1e7091f9d283_i1_5.jpg 0.0787463 0.00152096 0.996893 -2.9953 0.996503 -0.0281116 -0.078673 -14.4665 0.027905 0.999603 -0.00373071 1.36429 0 0 0 1

intrinsics_matrix 1074.56 0 639.97  0 1074.73 508.007  0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_0.png 03a8325e3b054e3fad7e1e7091f9d283_i2_0.jpg 0.903099 -0.28773 0.318784 -2.9979 0.429336 0.589254 -0.684435 -14.4541 0.0090875 0.754978 0.655687 1.39774 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_1.png 03a8325e3b054e3fad7e1e7091f9d283_i2_1.jpg 0.823448 0.377731 -0.423383 -2.98339 -0.567372 0.541901 -0.620026 -14.4553 -0.00477103 0.750774 0.660541 1.39764 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_2.png 03a8325e3b054e3fad7e1e7091f9d283_i2_2.jpg -0.0796293 0.669398 -0.738624 -2.97713 -0.996806 -0.0580366 0.0548659 -14.4684 -0.00614017 0.740634 0.671881 1.39742 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_3.png 03a8325e3b054e3fad7e1e7091f9d283_i2_3.jpg -0.903034 0.295597 -0.311691 -2.98537 -0.429522 -0.610609 0.665333 -14.4804 0.00634922 0.734697 0.678366 1.39729 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_4.png 03a8325e3b054e3fad7e1e7091f9d283_i2_4.jpg -0.823344 -0.369862 0.430474 -2.99988 0.567183 -0.56323 0.600895 -14.4792 0.0202075 0.738901 0.673511 1.39739 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_5.png 03a8325e3b054e3fad7e1e7091f9d283_i2_5.jpg 0.0797502 -0.661505 0.745688 -3.00614 0.996581 0.0367193 -0.0740088 -14.4661 0.021576 0.749041 0.662172 1.39761 0 0 0 1

intrinsics_matrix 1076.45 0 631.116  0 1077.19 509.202  0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d0_0.png 04e0d7506a0849009fabf4c10fa74a40_i0_0.jpg 0.850173 -0.370253 -0.374322 1.32069 -0.525592 -0.555036 -0.644739 -2.4959 0.0309546 0.74488 -0.666479 0.405217 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d0_1.png 04e0d7506a0849009fabf4c10fa74a40_i0_1.jpg -0.0302047 -0.674681 -0.73749 1.32766 -0.999434 0.0312638 0.0123318 -2.50867 0.0147369 0.737446 -0.675245 0.405386 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d0_2.png 04e0d7506a0849009fabf4c10fa74a40_i0_2.jpg -0.880632 -0.319222 -0.35012 1.32009 -0.473789 0.588126 0.655463 -2.52109 -0.00332313 0.743104 -0.669167 0.405267 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d0_3.png 04e0d7506a0849009fabf4c10fa74a40_i0_3.jpg -0.850661 0.340656 0.400409 1.30555 0.525688 0.558676 0.641508 -2.52075 -0.00516484 0.756197 -0.654324 0.404979 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d0_4.png 04e0d7506a0849009fabf4c10fa74a40_i0_4.jpg 0.0297349 0.645061 0.763552 1.29858 0.999496 -0.0276354 -0.0155765 -2.50798 0.0110534 0.76363 -0.645559 0.40481 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d0_5.png 04e0d7506a0849009fabf4c10fa74a40_i0_5.jpg 0.880141 0.28958 0.376156 1.30615 0.473817 -0.584484 -0.658692 -2.49555 0.0291131 0.757972 -0.651637 0.40493 0 0 0 1

intrinsics_matrix 1076.01 0 635.509  0 1076.38 511.999  0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d1_0.png 04e0d7506a0849009fabf4c10fa74a40_i1_0.jpg 0.853338 -0.0303201 -0.520474 1.31429 -0.520364 0.0120291 -0.853859 -2.50525 0.0321503 0.999467 -0.00551398 0.43837 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d1_1.png 04e0d7506a0849009fabf4c10fa74a40_i1_1.jpg -0.0241095 -0.0167139 -0.999569 1.31596 -0.99958 0.0164085 0.0238358 -2.50833 0.0160034 0.999725 -0.0171039 0.438411 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d1_2.png 04e0d7506a0849009fabf4c10fa74a40_i1_2.jpg -0.877724 -0.00611966 -0.479125 1.31414 -0.47916 0.00681266 0.877701 -2.51131 -0.00210672 0.999957 -0.00891331 0.438382 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d1_3.png 04e0d7506a0849009fabf4c10fa74a40_i1_3.jpg -0.853872 -0.00913176 0.520402 1.31064 0.520465 -0.00716229 0.853852 -2.51123 -0.00406951 0.999932 0.0108671 0.438313 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d1_4.png 04e0d7506a0849009fabf4c10fa74a40_i1_4.jpg 0.0235945 -0.0227381 0.999462 1.30897 0.999647 -0.0115411 -0.0238619 -2.50816 0.0120779 0.999674 0.0224564 0.438272 0 0 0 1
scan 04e0d7506a0849009fabf4c10fa74a40_d1_5.png 04e0d7506a0849009fabf4c10fa74a40_i1_5.jpg 0.877188 -0.033332 0.478985 1.31079 0.479193 -0.00194489 -0.877706 -2.50517 0.0301877 0.999442 0.0142651 0.438301 0 0 0 1

If I project 1 viewpoint using these files

dataset matterport
n_images 2358
depth_directory undistorted_depth_images
color_directory undistorted_color_images

intrinsics_matrix 1076.45 0 631.116  0 1077.19 509.202  0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_0.png 03a8325e3b054e3fad7e1e7091f9d283_i0_0.jpg 0.90525 0.275848 0.323155 -2.99825 0.42464 -0.612795 -0.666455 -14.4532 0.0141878 0.740533 -0.67187 1.33124 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_1.png 03a8325e3b054e3fad7e1e7091f9d283_i0_1.jpg 0.820534 -0.381542 -0.425615 -2.98374 -0.571596 -0.547236 -0.6114 -14.4543 0.000362848 0.744955 -0.667115 1.33115 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_2.png 03a8325e3b054e3fad7e1e7091f9d283_i0_2.jpg -0.0846661 -0.653405 -0.752259 -2.97748 -0.996408 0.0548167 0.0645317 -14.4674 -0.000928868 0.755021 -0.6557 1.33093 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_3.png 03a8325e3b054e3fad7e1e7091f9d283_i0_3.jpg -0.90513 -0.267872 -0.330125 -2.98573 -0.424975 0.591297 0.685393 -14.4794 0.0116044 0.760665 -0.649041 1.3308 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_4.png 03a8325e3b054e3fad7e1e7091f9d283_i0_4.jpg -0.820375 0.389515 0.418642 -3.00023 0.571259 0.525712 0.630309 -14.4783 0.0254291 0.756243 -0.653796 1.33089 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d0_5.png 03a8325e3b054e3fad7e1e7091f9d283_i0_5.jpg 0.0848417 0.661354 0.74526 -3.00649 0.996035 -0.076351 -0.0456356 -14.4651 0.0267202 0.746177 -0.665211 1.33112 0 0 0 1

intrinsics_matrix 1076.01 0 635.509  0 1076.38 511.999  0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_0.png 03a8325e3b054e3fad7e1e7091f9d283_i1_0.jpg 0.902625 -0.00853222 0.430341 -2.99332 0.430149 -0.017951 -0.902578 -14.4636 0.0154264 0.999802 -0.0125343 1.36432 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_1.png 03a8325e3b054e3fad7e1e7091f9d283_i1_1.jpg 0.82401 -0.00475809 -0.566553 -2.98983 -0.56657 -0.00416342 -0.824001 -14.4639 0.00156227 0.999979 -0.00612788 1.3643 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_2.png 03a8325e3b054e3fad7e1e7091f9d283_i1_2.jpg -0.0785585 0.00907021 -0.996867 -2.98832 -0.996908 -0.000537117 0.0785572 -14.467 0.000177485 0.999958 0.009083 1.36425 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_3.png 03a8325e3b054e3fad7e1e7091f9d283_i1_3.jpg -0.902493 0.0191241 -0.430278 -2.99031 -0.430517 -0.0106985 0.902519 -14.4699 0.0126569 0.999759 0.0178871 1.36422 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_4.png 03a8325e3b054e3fad7e1e7091f9d283_i1_4.jpg -0.823839 0.0153494 0.566614 -2.99379 0.5662 -0.0244859 0.823903 -14.4697 0.0265208 0.999582 0.0114802 1.36424 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d1_5.png 03a8325e3b054e3fad7e1e7091f9d283_i1_5.jpg 0.0787463 0.00152096 0.996893 -2.9953 0.996503 -0.0281116 -0.078673 -14.4665 0.027905 0.999603 -0.00373071 1.36429 0 0 0 1

intrinsics_matrix 1074.56 0 639.97  0 1074.73 508.007  0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_0.png 03a8325e3b054e3fad7e1e7091f9d283_i2_0.jpg 0.903099 -0.28773 0.318784 -2.9979 0.429336 0.589254 -0.684435 -14.4541 0.0090875 0.754978 0.655687 1.39774 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_1.png 03a8325e3b054e3fad7e1e7091f9d283_i2_1.jpg 0.823448 0.377731 -0.423383 -2.98339 -0.567372 0.541901 -0.620026 -14.4553 -0.00477103 0.750774 0.660541 1.39764 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_2.png 03a8325e3b054e3fad7e1e7091f9d283_i2_2.jpg -0.0796293 0.669398 -0.738624 -2.97713 -0.996806 -0.0580366 0.0548659 -14.4684 -0.00614017 0.740634 0.671881 1.39742 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_3.png 03a8325e3b054e3fad7e1e7091f9d283_i2_3.jpg -0.903034 0.295597 -0.311691 -2.98537 -0.429522 -0.610609 0.665333 -14.4804 0.00634922 0.734697 0.678366 1.39729 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_4.png 03a8325e3b054e3fad7e1e7091f9d283_i2_4.jpg -0.823344 -0.369862 0.430474 -2.99988 0.567183 -0.56323 0.600895 -14.4792 0.0202075 0.738901 0.673511 1.39739 0 0 0 1
scan 03a8325e3b054e3fad7e1e7091f9d283_d2_5.png 03a8325e3b054e3fad7e1e7091f9d283_i2_5.jpg 0.0797502 -0.661505 0.745688 -3.00614 0.996581 0.0367193 -0.0740088 -14.4661 0.021576 0.749041 0.662172 1.39761 0 0 0 1

It will project correctly as shown below.

I think it might be because the transformation matrix of each image in different viewpoints is independent rather than a relative global coordinate. So, I can't simply use the transformation matrix in the undistorted_camera_parameters to back-project the entire scan.