VTK capsule doesn't have the texture coordinate defined

[zachfang]

What happened?

This issue was discovered in the RenderRpc project when we wanted to set up an example scene to test the RenderEngine rendering. Due to the lack of texture coordinate, the capsule is not properly textured given a texture map either when loading in code or via an SDFormat file.

For example:

  const Capsule capsule(0.05, 0.1);
  scene_graph->RegisterAnchoredGeometry(
      source_id,
      MakeInstance(
          capsule, Vector3d(x, 0.25, 0),
          Material(texture_path, label_value), "texture_capsule"));

    <link name="textured_capsule">
      <pose>0.25 0.25 0 0 0 0</pose>
      <visual name="textured_capsule_visual">
        <geometry>
          <capsule>
            <radius>0.05</radius>
            <length>0.1</length>
          </capsule>
        </geometry>
        <material>
          <drake:diffuse_map>4_color_texture.png</drake:diffuse_map>
        </material>
      </visual>
    </link>

Technically, the capsule on the right should have the same texture as other primitives, but the default white color is rendered instead.

Version

No response

What operating system are you using?

Ubuntu 20.04

What installation option are you using?

No response

Relevant log output

No response

[jwnimmer-tri]

Do we know yet where in the stack this problem happens? VTK's GLTF exporter?

[zachfang]

I haven't dug in too much as I only treat it as a known issue during the RenderRpc development, but @SeanCurtis-TRI might have more insight. I can try to fix it after my last RenderRpc push.

[SeanCurtis-TRI]

Which render engine produced the image?

Ultimately, this is a per-render engine issue. Each one is obliged to provide textures coordinates for the primitives. The capsule was added without. I presume VTK should be able to activate them relatively easily. They'll have to be done by handle in GL. For a rpc render engine, they all have the unique responsibility to define what it means to "render a capsule".

[zachfang]

The capsule is not properly textured from both RenderEngineVtk and RenderEngineGltfClient as they use VTK under the hood. OK, I can look into fixing the VTK part first.

[SeanCurtis-TRI]

It looks like you're in trouble.

vtkCapsuleSource doesn't produce "TCoord" data in the polygon data. You can contrast this with what vtkCubeSource does. It seems like this is a change/patch to VTK in order to get texture coordinates on the capsule.

[bhaq-tri]

@SeanCurtis-TRI, What is the best way to implement the change? Can we patch a fix and use it within Drake? Or should we ping VTK with issue and request them to fix?

[SeanCurtis-TRI]

I strongly suspect that you can patch the CreateVtkCapsule() and post-hoc add UV coordinates while we wait for VTK updates and our ability to consume those changes. I haven't looked into the specific details, but it seems incredibly that we can't shove UV coords into that data structure after the fact.

[zachfang]

FYI, @svenevs will provide some updates on how to patch it in VTK and how to incorporate the fix into Drake once he nails down the details.

[svenevs]

Sorry for the delay here, the most "straightforward" path to getting this would be to create a new class and contribute it to VTK by submitting a merge request to the VTK GitLab. GitLab contributions are similar to GitHub (you will be able to sign into the kitware gitlab using your GitHub credentials if you prefer), you fork the repository, push a branch and open up a merge request. CC me (@svenevs there too).

• You would be wanting to create a new vtkTexturedCapsule class.

  • You can model your initial files vtk/Filters/Sources/vtkTexturedCapsule.h and vtkTexturedCapsule.cxx after the textured sphere files vtkTexturedSphereSource.h and vtkTexturedSphereSource.cxx.
  • In the .cxx file the texture coordinates are operated with newTCoords in the vtkTexturedSphereSource::RequestData method.
  • Add vtkTexturedCapsule to the list of classes in the associated CMakeLists.txt

  • When you have the prototype ready and a cmake build of VTK configured / built locally (in say ~/vtk_build), you can test your changes by modifying the texture plane example

    • Download TexturePlane.cxx and CMakeLists.txt into a test folder (placed next to each other). To avoid needing to install and reconfigure each time, if you configure this test project with -DVTK_DIR=/abs/path/to/vtk_build_directory then find_package(VTK ...) will be able to use your build tree. This will enable you to iteratively test your new texture coordinates by (a) update VTK source code and re-build and then (b) recombile the test project. You will not need to install VTK each time to test.

    • Make sure to #include <vtkTexturedCapsule.h> and change the line vtkNew<vtkPlaneSource> plane; to be vtkNew<vtkTexturedCapsule> plane; (leaving it called plane to avoid refactoring everything).

    • Do be sure to use good test patterns during development, such as this one and/or checkerboards etc, the seams along the caps and cylinder side are going to be the areas of interest:

      

    • One thing of note is to in your example test "texture plane" program, try out vtkNew<vtkTexturedSphere> (and #include it), set the resolution of your sphere to something acceptable -- the defaults make it seem like the sphere has bad texture coordinates, but really there's just very very few triangles and it looks bad. A similar resolution for the capsule for testing is encouraged, you don't want to waste a bunch of time thinking your tcoords are broken when really it's just getting scrunched because too few triangles.

• Seeing the implementation details above, with the textured plane example, you may be tempted to create geometry and add things after the fact using vtkPointData *pointData = planeMapper->GetInput()->GetPointData(); -- calling SetTcoords with a new float array or modifying the buffers directly. DO NOT DO THIS, you are directly modifying the input to the mapper (and hence the output of a filter) outside of the pipeline. If the pipeline reexecutes then you have to track this and update the pipeline again.
• Given the above point about filters and pipeline editing being unsafe in this scenario, while I have not been able to look into getting this to work, the answer is to use "programmable filters". vtkProgrammableAttributeDataFilter. There are examples to be found, including the test MultidimensionalSolution.py. Also a related blog post: https://berkgeveci.github.io/2014/09/02/programmable-filter/

My advice would be to just get the texture coordinates working in VTK. I will be able to help make that available in the drake-vtk build if that is the route taken (basically, once the VTK patch is ready you can hand that off to me and I'll create new binaries after adding the patch to tools/workspace/vtk/). I hope the above helps get you unstuck, the math behind the texture coordinates should be the hardest part.

[SeanCurtis-TRI]

Whoooooo boy! That's going to be a lot to digest. I'll have to come back to that...

The good news is that the math for the texture coordinates is the easy part. Ping me when we get around to it. :) What I might do is add the texture coordinates to the GL renderer and that can serve as a basis for the math. And then everything else will be working with VTK.

[SeanCurtis-TRI]

Also another solution is to write the vtkTexturedCapsuleSource completely in Drake. When we first introduced the capsule, we vendored it from a third party and simply brought it into drake until it was eventually transported to VTK. Doing it in Drake allows us a Drake developer to use the Drake ecosystem to test for correctness and exercise and shortens the path to having the behavior available in Drake. Once it gets pushed back to vtk, we can then update Drake's internals to delete our bespoke version and use that contained in the library.

[svenevs]

Also another solution is to write the vtkTexturedCapsuleSource completely in Drake.

Yes, this is the best path forward -- good call! I thought you would have to work with the .tar.gz build to do it in drake first, but this will be way better... Add a new type for drake in the desired location, starting with the base implementation of the existing vtkCapsuleSource and just implement RequestData such that it also adds the desired texture coordinates. The rest of VTK should detect and use it as expected.

When something desirable is achieved I can help get it into VTK / the drake VTK build fairly quickly.

[bhaq-tri]

PR open #18547 and Gitlab VTK.

[svenevs]

We've got a VTK dev looking into this next week! There might be a different way to get the texture coordinates in using a different strategy, we will update soon!

[jwnimmer-tri]

@svenevs Sounds great! Please post progress updates and/or questions to this ticket. (Especially if a MR gets opened.)

[sankhesh]

I've been looking into this and am adding the functionality in a more generic form. The idea is to add an API on the vtkCylinderSource to have hemispherical capping with tcoords. This should also make the geometry cleaner. See the images below.

[SeanCurtis-TRI]

To make sure I understand what I'm seeing there -- that's the wireframe applied as a texture map, right?

[sankhesh]

Yes, rendering cell edges.

[BetsyMcPhail]

VTK MR: https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10531

[sankhesh]

The VTK merge request has been merged.

[jwnimmer-tri]

@SeanCurtis-TRI @zachfang if you like, instead of waiting for the next VTK release, it's plausible you could to cherry-pick the patch into https://github.com/RobotLocomotion/drake/tree/master/tools/workspace/vtk_internal/patches in the meantime, assuming that there aren't any too-difficult merge conflicts.

[SeanCurtis-TRI]

I'll take a look at cherry-picking.

[jwnimmer-tri]

@SeanCurtis-TRI if / when you have any progress, please post a pingback here.

[SeanCurtis-TRI]

Will do.

[jwnimmer-tri]

Plausibly also @zachfang could work on cherry-picking this, if you like.

[sankhesh]

@jwnimmer-tri @SeanCurtis-TRI The change is available in VTK 9.3.0-rc2

[jwnimmer-tri]

We can't upgrade our version until #19965 is fixed, so for now it'll need to be cherry-picking.