How to render a mesh without texture using habitat-sim

[fangchuan]

Hi all, I am able to render rgb/depth images from textured meshs in Replica-dataset, but now I hope to render image with illumination turned on but surface texture turned off. I have tried several methods to reach the goal, but it failed. The version of code I am using is branch main tag v0.2.0， like the picture below. And I follow the replica-dataset instruction to load data and render images.

First, I set all vertexts' color to the user-defined value during PTexMeshData::parsePLY, specificly, I revise the follwing lines https://github.com/facebookresearch/habitat-sim/blob/59456b9efc012f2f08c324b97742570bdcf8f041/src/esp/assets/PTexMeshData.cpp#L777-L779

into

    if (colorDimensions != 0u) {
      memcpy(meshData.cbo[i].data(), &nextBytes[colorOffsetBytes], colorBytes);
      memset(meshData.cbo[i].data(), uint8_t(192), colorBytes);
    }

In this way, the rendering image is still textured, so I thought it could be caused by PTex files loaded in the process of PTexMeshData::load.

So, in the next step, I create a fake_image_data in the process of PTexMeshData::uploadBuffersToGPU, revising the following codes https://github.com/facebookresearch/habitat-sim/blob/59456b9efc012f2f08c324b97742570bdcf8f041/src/esp/assets/PTexMeshData.cpp#L916-L941 into

    Cr::Containers::Array<const char, Cr::Utility::Directory::MapDeleter> data =
        Cr::Utility::Directory::mapRead(hdrFile);
    // divided by 6, since there are 3 channels, R, G, B, each of which takes
    // 1 half_float (2 bytes)
    const int dim = static_cast<int>(std::sqrt(data.size() / 6));  // square
    CORRADE_ASSERT(dim * dim * 6 == data.size(),
                   "PTexMeshData::uploadBuffersToGPU: the atlas texture is not "
                   "a square", );

    char* fake_data = (char*)malloc(data.size());
    memset(fake_data, uint8_t(192), data.size());
    Cr::Containers::Array<const char, Cr::Utility::Directory::MapDeleter>
        fake_img(fake_data, data.size());

    // atlas
    // the size of each image is dim x dim x 3 (RGB) x 2 (half_float), which
    // equals to numBytes
    Magnum::ImageView2D image(Magnum::PixelFormat::RGB16F, {dim, dim},
                              fake_img);

    renderingBuffers_[iMesh]
        ->atlasTexture.setWrapping(Magnum::GL::SamplerWrapping::ClampToEdge)
        .setMagnificationFilter(Magnum::GL::SamplerFilter::Linear)
        .setMinificationFilter(Magnum::GL::SamplerFilter::Linear)
        .setStorage(
            Magnum::Math::log2(image.size().min()) + 1,  // mip level count
            Magnum::GL::TextureFormat::RGB16F, image.size())
        .setSubImage(0,   // mipLevel
                     {},  // offset
                     image)
        .generateMipmap();

Unfortunately, I get an empty rendering image (totally black) in the end.

Could you help me figure out an appropriate method to achive the goal ? @aclegg3 @0mdc , I'm looking forward to your help. Thanks a lot!

[aclegg3]

Hey @fangchuan, Haven't had time to dig into this. The PTex support is a separate code path and shader from any other asset support. I think you are on the right track here.

My first two approaches would be:

1. Edit the pipeline to remove the texture info programmatically (your approach)
2. Try to strip or clear the texture information from the asset before loading. For example, replace the texture images with images containing the flat color you want.

[fangchuan]

Hey @fangchuan, Haven't had time to dig into this. The PTex support is a separate code path and shader from any other asset support. I think you are on the right track here.

My first two approaches would be:

1. Edit the pipeline to remove the texture info programmatically (your approach)
2. Try to strip or clear the texture information from the asset before loading. For example, replace the texture images with images containing the flat color you want.

Thanks for your reply. Well, there maybe some questions about your two suggestions:

1. I do edit the pipeline to avoid loading texture, but I only get a black rendering image from color_sensor in the end;
2. Im not sure is there any specific format used by texture files: xxx-color-ptex.hdr, these files' sizes are huge and I dont know if they are organized in line with ordinary texture images which use uv coordinate.