3D mesh in VisPy

[asnt]

(Suggested) Ticket to keep track of improvements for 3D mesh rendering (MeshVisual).

Data representation

• [x] [PR #1462] Adapt internal representation of arrays for simplifying filter creation.

Rendering

• Shading:
  • [x] [PR #1463] Shading filter (Phong reflection model).
  • [x] [PR #2028] Some fixes.
  • [x] [PR #2029] Identify/expose Phong parameters.
  • [ ] [PR #2091] Fix computation of eye direction.
• Material:
  • [x] [PR #1444] Texture filter: texture mapping for a single material.
  • Normal mapping (#1615).
  • Support multiple materials (#1984).
• Edges/wireframe (#1650 #1508):
  1. [x] [PR #1689] Single-pass wireframe from barycentric coordinates [1, 2]: (+) line width, antialiasing; (-) line width not depth-dependent (units in screen-space only?), requires expanded geometry (i.e. vertex buffer without index buffer, larger memory footprint)
  2. [ ] Single-pass wireframe from vertex positions [wire1a, wire1b]: (+) line modulation with depth and/or relative position between two vertices (units in world and screen space) (-) requires more vertex attributes
  3. [ ] [PR #1510] drawing edge segments in mode 'lines': (+) simple method (-) requires a separate visual (similar to MeshNormals), no guarantee on configurable line width (dependent on the opengl driver), no antialiasing
  4. [ ] Using glPolygonMode [2]: similar to above, (-) only in gl2+ (not in gles 2/3), control of line width not guaranteed, no anti-aliasing
  5. [ ] Using geometry shaders [nVidia's Solid Wireframe, 2]: (+) depth-aware width, less data in vbo (-) performance cost of geometry shader
• Transparency (#1345):
  1. [ ] [PR #2039] Weighted blended order-independent transparency [oit-weighted]: (+) fast (-) not exact, artefacts for large alpha (best with alpha < 0.25) EDIT: No, the weighted version works for any alpha (only the simpler blended OIT without depth weights darkens the colors for alpha > 0.25).
  2. [ ] Depth peeling: (+) exact (-) expensive (N render passes for N transparent layers)
  3. [ ] Dual-depth peeling: (+) exact, less expensive than depth peeling (if gl3+) (-) expensive (N/2 + 1 render passes for N transparent layers), requires gles3+/gl3+ for multiple render targets (otherwise N passes)
• Batch rendering:
  • Render many copies of a single geometry (sphere, box, arbitrary mesh) with varying attributes per copy (scale, position, color...).
  • Note: Same idea as the vispy "collections" for 2D objects. Check if the concept/implementation can be reused.
  • Note: With gl3.1+/gles3+ this can be done efficiently with instanced rendering.
  • Related: #1071
  • [ ] [PR #2008] PointSet: SpheresVisual drawn in mode "points" (+) efficient (-) limited sphere size?
  • [ ] MeshesVisuals for an arbitrary mesh geometry (Examples for boxes in #1814, #1820.)

Camera

• Implement autofitting the camera frustum to the objects in the scene.

Lights

• discussion on an API in #553
• multiple light sources in a scene (a specific node for a light source in the scene graph?)
• different types of light sources (ambient, point, directional...)
• headlight (light attached to a camera following the viewing direction)

Loading

• (using meshio #1817 could help address some (all?) of the items below)
• Handle attributes on face corners instead of assuming vertex attributes (issue of #1236, it seems)
• Load material/texture, e.g. from obj/mtl (#1468)
  • Handle multiple materials.
• Add flexibility to the data structure returned from a loader (currently fixed tuple (vertices, faces, normals, texcoords) assuming vertex attributes only)
• More mesh formats (?): stl (#1594), .off, .ply, .gltf
• Handle the non-standard color vertex attribute in .obj files (#1513), i.e. parse vertex lines as v x y z [r g b] with optinal rgb color. This seems is a common enough need for scientific visualisation to be worth adding. Meshlab does that.

[djhoese]

Maybe vispy 0.7 is the release of mesh updates. The issues from @infosisio are making me realize how much functionality is missing from the MeshVisual and related utilities for displaying .obj (and related) files. I don't personally use Meshes directly in my own work.

@asnt Would you mind joining us on gitter so we can discuss a plan of attack for getting all of these things merged or resolved. I know you have a lot of PRs that are overdue and I'd like get them in, but might need some guidance.

[asnt]

@djhoese v0.7 as target would be great. I'll give a hand where I can.

[sofroniewn]

@djhoese I was wondering what the status of some of these PRs is - namely #1463 - which provides better shading for mesh visuals (and which depends on #1462). I havn't tried any of them out yet, but I think napari would benefit from some nicer shading options for 3D meshes, and #1463 seems like it will provide that.

I am happy to help review / test etc. as needed - just let me know if I can be of any use.

[djhoese]

@sofroniewn A lot of the PRs made by @asnt are very close to being ready to be merged. I have a couple comments in them that I think I'm still waiting on a response for from @asnt.

I appreciate the willingness to help out. If you want to review the various PRs from @asnt, figure out what makes sense, and point out parts that you have concerns about that would be amazing. It is a little difficult to keep track of the various parts of these PRs since I don't really use Meshes directly myself.

[sofroniewn]

Sounds good @djhoese - I'll look at #1462 now as it seems like that is a foundational one

[asnt]

Great to see interest and thanks for your reviews, @sofroniewn @djhoese. I haven't forgotten about these PR's and am still willing to bring them to an acceptable state. I'll try my best in the coming days ;-)

[djhoese]

I checked a couple of the boxes near the top of the main list that PRs were merged for. Feel free to review this @asnt and let me know if I missed any that you've resolved.