How to generate NMR like dataset for my own custom Image-3Dmodel pair

[jay-thakur]

Hi,

I have downloaded dataset (b. Shapenet for 2D supervised model).

I tried to load & see cameras.npz dataset, it looks like this - these are all 4*4 numpy array

@m-niemeyer Could you please explain this & how to generate dataset like this for my own image 3D model.

Thanks, Jay

[rodrigodzf]

A similar follow-up question, what is the function of the scale matrix? is the K-matrix already scaled?

[m-niemeyer]

Hi @jay-thakur, @rodrigodzf , thanks a lot for your interest in the project! In think both questions are pointing at what the matrices are, here my explanations:

world_mat: This is the extrinsics 4x4 matrix, often referred to as "Rt". When applying this matrix to a point, you transform this point from world space to camera space. camera_mat: This is the intrinsics 4x4 matrix, often referred to as "K". The only special thing we do is that we define the final pixel coordinates to be in the interval [-1, 1]. In contrast, the ordinary definition is to have pixel values in [0, W] and [0, H], where (H, W) are the image dimensions. (We use this special definition because we think it fits better the Pytorch framework.) scale_mat: We introduce this additional 4x4 matrix for the case when the objects are not centred at the world origin and not in the unit cube. This matrix transforms a 3D point from the "object-centric" space where the object is centred at the origin and in the unit cube to the original world space. You can see that this matrix is just the identity for e.g. the ShapeNet datasets. We only use this matrix for the DTU dataset, where objects are not centred. Have a look at this closed issue for more information.

[rodrigodzf]

Thank you for the explanation! As suggested in the other closed issue, maybe would be good idea to put this information in the readme since I imagine others will have the same question in the future

[m-niemeyer]

Thanks for the feedback @rodrigodzf , I will try to do it soon!

[jay-thakur]

Hi @m-niemeyer ,

I have my own 3D model. Below is my viewpoints for 3D model using open3d library.

{
    "class_name" : "PinholeCameraParameters",
    "extrinsic" : 
    [
        1.0,
        -0.0,
        -0.0,
        0.0,
        0.0,
        -1.0,
        -0.0,
        0.0,
        0.0,
        -0.0,
        -1.0,
        0.0,
        -0.0,
        -0.061849594116210938,
        117.97353744733314,
        1.0
    ],
    "intrinsic" : 
    {
        "height" : 1050,
        "intrinsic_matrix" : 
        [
            909.3266739736606,
            0.0,
            0.0,
            0.0,
            909.3266739736606,
            0.0,
            959.5,
            524.5,
            1.0
        ],
        "width" : 1920
    },
    "version_major" : 1,
    "version_minor" : 0
}

How to rescale camera_mat and world_mat for 64*64 size projection?

I hope you got my question.

Thanks, Jay