how to get pixel-corresponding depth skybox?

[seigeweapon]

Although the dataset is awesome, I find it very inconvenient not having a pixel-corresponding depth map in skybox form. It provides only color skybox. I tried using PanoBasic to stitch that out of the undistorted color/depth images, with camera extrinsics from undistorted_camera_parameters. But the result is poor. Then I tried using commercial stitching software like PTGUI, but only find it's hard because of very small overlapping area.

Anybody run into this like me? Help me please!

[HalleyJiang]

@seigeweapon, Hi seigeweapon, I also have the same problem. Have you solved it?

[seigeweapon]

No. I'm still having problem. I put my code and result here, hopefully someone can point out what's wrong.

` add_path;

img_w = 1280;
img_h = 1024;

intrinsics = [
1075.72 0 636.877  0 1075.66 515.013  0 0 1
1075.94 0 629.693  0 1076.09 501.638  0 0 1
1074.61 0 630.692  0 1074.93 523.124  0 0 1
];

data_path = 'D:\data\matterport\17DRP5sb8fy';
scene_id = '00ebbf3782c64d74aaf7dd39cd561175';

sepImg = [];
rts = [
-0.73677 -0.438446 -0.514718 3.52064 -0.675941 0.496262 0.544819 -0.501891 0.0165615 0.749326 -0.661995 1.54282 0 0 0 1
-0.953749 0.211332 0.213781 3.50647 0.300171 0.631344 0.715054 -0.505191 0.0161444 0.746152 -0.665579 1.54289 0 0 0 1
-0.216901 0.653201 0.725453 3.49653 0.976122 0.136149 0.169259 -0.494569 0.0117899 0.744843 -0.667135 1.54292 0 0 0 1
0.736908 0.445283 0.508615 3.50076 0.675947 -0.494116 -0.546758 -0.480646 0.00785256 0.746707 -0.665106 1.54288 0 0 0 1
0.953849 -0.2045 -0.219891 3.51493 -0.300173 -0.629173 -0.716964 -0.477347 0.00826982 0.749881 -0.661521 1.54281 0 0 0 1
0.216975 -0.64635 -0.731542 3.52487 -0.976096 -0.133961 -0.171149 -0.48797 0.0126244 0.75119 -0.659965 1.54278 0 0 0 1

-0.729995 0.00798594 -0.683406 3.51324 -0.683354 0.00847387 0.730038 -0.493776 0.011621 0.999932 -0.000728289 1.57609 0 0 0 1
-0.956787 0.0123707 0.290529 3.50983 0.290574 0.00201265 0.956851 -0.49457 0.011252 0.999922 -0.00551992 1.5761 0 0 0 1
-0.226736 0.0089675 0.973915 3.50744 0.973932 -0.00501523 0.226786 -0.492015 0.00691798 0.999948 -0.00759623 1.57611 0 0 0 1
0.730089 0.00117961 0.683352 3.50846 0.683346 -0.00558174 -0.730073 -0.488665 0.00295295 0.999984 -0.00488087 1.5761 0 0 0 1
0.956842 -0.00320491 -0.290592 3.51187 -0.29059 0.000879651 -0.956847 -0.487872 0.00332209 0.999995 -8.92584e-05 1.57609 0 0 0 1
0.226765 0.000198574 -0.97395 3.51426 -0.97392 0.0079074 -0.226756 -0.490427 0.00765625 0.999969 0.00198689 1.57608 0 0 0 1

-0.728354 0.466698 -0.50169 3.52095 -0.68513 -0.485575 0.542967 -0.501795 0.00979302 0.739196 0.673419 1.60931 0 0 0 1
-0.957507 -0.186404 0.220078 3.50678 0.288256 -0.643483 0.709111 -0.505095 0.00943564 0.742417 0.669871 1.60938 0 0 0 1
-0.229106 -0.649703 0.724843 3.49683 0.973388 -0.156822 0.167099 -0.494473 0.00510657 0.743837 0.668342 1.60941 0 0 0 1
0.728431 -0.45989 0.507828 3.50106 0.685118 0.487735 -0.541043 -0.48055 0.00113495 0.742035 0.67036 1.60938 0 0 0 1
0.957545 0.193218 -0.213947 3.51523 -0.288278 0.645618 -0.707159 -0.477251 0.00149246 0.738813 0.673908 1.60931 0 0 0 1
0.229118 0.656498 -0.718691 3.52517 -0.973381 0.158939 -0.165128 -0.487873 0.00582165 0.737394 0.675438 1.60928 0 0 0 1
];

idx = 1;
for j = 1:3
    intrinsic = transpose(reshape(intrinsics(j,:), [3,3]));
    fx = intrinsic(1,1);
    fy = intrinsic(2,2);
    cx = intrinsic(1,3);
    cy = intrinsic(2,3);
    crop_x_start = 1;
    crop_x_end = img_w;
    crop_y_start = 1;
    crop_y_end = img_h;
    if cx * 2 <= img_w
        crop_x_end = round(cx*2);
    else
        crop_x_start = round(1 + cx*2 - img_w);
    end
    if cy * 2 <= img_h
        crop_y_end = round(cy*2);
    else
        crop_y_start = round(crop_y_start + cy*2 - img_h);
    end

    hfov = 2 * atan(((crop_x_end - crop_x_start + 1) / 2 / fx));

    for i = 1:6
        img = im2double(imread(fullfile(data_path, 'undistorted_color_images', ...
            sprintf('%s_i%d_%d.jpg', scene_id, j-1, i-1))));

        % get yaw roll pitch from rt
        rt = transpose(reshape(rts((j-1)*6+i,:), [4,4]));
        r = rt(1:3,1:3);
        yrp = num2cell(rotm2eul(r));
        [yaw roll pitch] = yrp{:};
        pitch = pitch - 1.57;
        fprintf("%f %f %f %f\n", yaw, roll, pitch, hfov);

        % rotate image
        img = imrotate(img, -roll * 180 / pi, 'bilinear', 'crop');

        % crop image so that camera center is image center
        img = img(crop_y_start:crop_y_end,crop_x_start:crop_x_end,:);

        sepImg(idx).img = img;
        sepImg(idx).vx = -yaw;
        sepImg(idx).vy = pitch;
        sepImg(idx).fov = hfov;
        sepImg(idx).sz = size(sepImg(idx).img);
        idx = idx + 1;
    end
end

panocolor = combineViews( sepImg, 2048, 1024 );
figure; imshow(panocolor);

`

[manurare]

@seigeweapon @HalleyJiang Any update on this? I rendered the depth for skybox images using blender but it is still not perfectly aligned. Does anybody know the exact poses for the skybox images?

[HalleyJiang]

The authors even could not reply about the exact poses for the skybox images. We can just approximately align the depth to the panoramas from the skybox images. It's ok for panorama depth estimation. Please refer to our work on preprocessing Matterport3D(https://github.com/alibaba/UniFuse-Unidirectional-Fusion/blob/main/UniFuse/Matterport3D/README.md).

[manurare]

Hey Thanks for the answer. On the repo you sent, there is a function called stitch in stitching_Matterport3D.m. There is no implementation of that function and matlab doesnt find it. Am I missing anything?

[HalleyJiang]

You need to download PanoBasic first. Please carefully read the README.

[manurare]

I did it. But there is not stitch function anywhere and raises an error

[HalleyJiang]

We missed some .m files. We have added them. Please try again.