Inconsistent channel in texture map

[jh27kim]

If you do not know the root cause of the problem / bug, and wish someone to help you, please post according to this template:

🐛 Bugs / Unexpected behaviors

NOTE: Please look at the existing list of Issues tagged with the label 'bug`. Only open a new issue if this bug has not already been reported. If an issue already exists, please comment there instead..

Instructions To Reproduce the Issue:

Please include the following (depending on what the issue is):

1. Any changes you made (git diff) or code you wrote No change has been made

2. The exact command(s) you ran:

   io = IO()
   io.register_meshes_format(MeshGlbFormat())
   with open(glb_path, "rb") as f:
   mesh = io.load_mesh(f, include_textures=True)

3. What you observed (including the full logs): renderer/mesh/textures.py throws an error when texture image channel is not consistent. I used a mesh from Objaverse so I guess the mesh should be sound. Any advice ?

[bottler]

I don't know what it means that there is a map with a different number of channels. Perhaps it is a type of data pytorch3d cannot use and should ignore. Can you share the json from the beginning of the file, e.g. via pastebin or uploading it here?

[jh27kim]

This is the mesh which I found the problem above. Maybe it's due to mesh's unsoundness. I will also check if other meshes behave similary. updated_mesh.glb.zip

Plus, I found another issue occuring at https://github.com/facebookresearch/pytorch3d/blob/f34104cf6ebefacd7b7e07955ee7aaa823e616ac/pytorch3d/io/experimental_gltf_io.py#L422

is there any workaround for this issue ?

[bottler]

There are two texture map images, one is RGBA and the other is 2 channel. I haven't seen such 2 channel images, but it makes sense they could be greyscale and transparency.

So in pytorch3d/io/experimental_gltf_io.py around line 271 can you try replacing

        im = torch.from_numpy(self._access_image(image_index))[:, :, :3]

with

        im = torch.from_numpy(self._access_image(image_index))
        C = im.shape[2]
        if C == 4:
            im = im[:, :, :3]
        elif C == 2:
            im = im[:, :, :1].expand(-1, -1, 3)

I think that's the right thing. Looking at the image in https://gltf-viewer.donmccurdy.com/ , it looks like the body of the fridge / refrigerator is colorful and the door might only be grayscale.

If this works, let's make the change to pytorch3d.

(Alternatively, you can use experimental_gltf_io.load_meshes to get the separate parts of the scene into PyTorch3D, and then make the textures compatible yourself: You will see one of the five meshes only gets two channels.)