Add GLTFLib parsing abilities to PyRender

aelzeiny commented 2 years ago

Thank you

First off, I really, really like your library. Code is readable, and the structure is GLTF compliant which makes development kinda nice. Thanks for making this!

Example use

GLTF lib parses GLTF or GLB. It also resolves and reads all uris and file resources.

import gltflib
import pyrender

# Entire Scene
gltf = gltflib.GLTF.load('./some_model.gltf')
scene = pyrender.Scene.from_gltflib_scene(gltf)

# Just one Mesh
specific_mesh = gltf.meshes[0]
mesh = pyrender.Mesh.from_gltflib(specific_mesh, gltf)

Problem/Motivation

Okay, onto the PR. PyRenderer is based off of the GLTF specification. That means that one day it should have support for Skinning, Morph Targets (aka Blend Shapes), and Animations. Unfortunately, the TriMesh library does its one job really well, but it will never support all the features in GLTF2. I can't imagine that a meshing library would start to parse animations.

Solution

So in my opinion, the next step for this project is to start reading from GLTF and GLB files directly. Fortunately, we don't have to reinvent the wheel, GLTFlib parses and types these files for us to a certain extent. Converting gltflib.GLTF to a pyrender.Scene does take some knowledge on how buffers, buffer views, and accessors work, and that's the work that this PR provides. The result is nearly identical to reading through TriMesh. A little bit of code in this PR has been refactored from TriMesh repo, especially in regards to reading binary buffer data. In the future, if this PR goes well, I'm going to add Morph Targets + weights support.

Correctness

In order to ensure correctness, I compared the new GLTF parsing libraries with TriMesh. I tested this in an adhoc-fashion for all the models under Kronos open-source models. We are about 100% compatible with GLTF and 99% compatible with Trimesh. Every numpy array for position, UV Textures, Normals, etc are exactly the same. The one exception is Materials, where the new implementation is more accurate than TriMesh. It turns out that the best-effort conversion from GLTF -> TriMesh -> Pyrender actually changes the original material. Also, trimesh does not support or parse targets

aelzeiny commented 2 years ago

Hey Matt, I haven't added sufficient documentation or test-cases to this PR. Since we never had a conversation, I wanted to first gauge interest in going with this direction. With your thumbs up I can go the rest of the distance on this (i.e: remove type-hints and follow repo's style more closely).

Best Regards, Ahmed

mmatl commented 2 years ago

This is awesome! Will review soon and get it merged up. Thanks so much for the PR!

aelzeiny commented 2 years ago

I'm glad you like it! I'll keep at this then, add more test cases, and follow the code-style more carefully. Please don't merge quite yet.

aelzeiny commented 2 years ago

Hoollly crap. The last 5% of this PR is absolutely kicking my butt. I'm going to rip this down for now until it's ready. Sorry for the delay.

mmatl / pyrender