[branch "rendermesh-refactor"] ConvertToRenderScene() failing

tellypresence commented 3 months ago

I'm looking at migrating from dev to rendermesh-refactor branch.

Previously I was able to load 4 of the 15 models from apple (toy drummer, toy car, robot, guitar)

But now ConvertToRenderScene fails (previously worked on dev) whether I provide a RenderSceneConverterEnv or Stage

converter.ConvertToRenderScene(env, &render_scene);
converter.ConvertToRenderScene(stage, &render_scene);

(Understand this branch may not yet be stable in which case these errors would be expected, apologies if that is the case)

syoyo commented 3 months ago

Yes, rendermesh-refactor is current active working branch and soon-to-be merged into dev(I'd like to finish it by this month(2024/Apr)). This branch does mesh triangulation, material/texture conversion(includes geomsubset material), etc. suited for rendering USD(Z) on OpenGL/Vulkan graphics API.

Some errors are expected. For a while, you can try running tydra_to_renderscene example https://github.com/syoyo/tinyusdz/tree/dev/examples/tydra_to_renderscene and how it goes with USDZ files. (Reporting issues are much appreciated to add a fix to rendermesh-refactor branch and polish Tydra API)

NOTE: When rendermesh-refactor branch is merged into dev, it will make you easily write a viewer app(including Android example viewer).

tellypresence commented 3 months ago

Some information when using rendermesh-refactor branch, I added some logging and tried with a few models, hopefully some of the data is useful for troubleshooting

Using `converter.ConvertToRenderScene(env...)`

Working models

Models that successfully complete (NOTE: face vertex indices are invalid as indicated in later post below)

suzanne.usdc
compsognathus.usdz
Furriers_guild_chest_in_Myslenice.usdz (issue #109)
untitled.usdz (issue #115)
Netsuke_-_Shoki_capturing_an_oni.usdz (issue #116)

~Failing models~

(Update: the 4 models from apple now load following fix) Model toy_car.usdz fails in VisitPrimsRec(): Visit function returned an error for Prim /toy_car/toy_car_realistic_lod0 (id 15)

render-data (C++)   D  ConvertToRenderScene(env): intro: scene0x6d7c6fe3a8
scene-access (C++)  D  VisitPrims(): intro (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 1734)
scene-access (C++)  D  VisitPrimsRec(): intro: root_abs_path: /toy_car (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 290)
scene-access (C++)  D  VisitPrimsRec(): intro: root_abs_path: /toy_car/toy_car_realistic_lod0 (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 290)
scene-access (C++)  E  Visit function returned an error for Prim /toy_car/toy_car_realistic_lod0 (id 15) (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 305)
scene-access (C++)  W  VisitPrims(): !VisitPrimsRec(root_abs_path), root_abs_path: /toy_car (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 1752)
render-data (C++)   E  ConvertToRenderScene(env): VisitPrims fail (file: <...>/tinyusdz_repo/src/tydra/render-data.cc, line: 4897)

Model fender_stratocaster.usdz fails with Visit function returned an error for Prim /fender_stratocaster/StratocasterStand (id 80)

render-data (C++)   D  ConvertToRenderScene(env): intro: scene 0x6d8bdb18d8
scene-access (C++)  D  VisitPrims(): intro (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 1734)
scene-access (C++)  D  VisitPrimsRec(): intro: root_abs_path: /fender_stratocaster (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 290)
scene-access (C++)  D  VisitPrimsRec(): intro: root_abs_path: /fender_stratocaster/StratocasterStand (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 290)
scene-access (C++)  E  Visit function returned an error for Prim /fender_stratocaster/StratocasterStand (id 80) (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 305)
scene-access (C++)  W  VisitPrims(): !VisitPrimsRec(root_abs_path), root_abs_path: /fender_stratocaster (file: <...>/tinyusdz_repo/src/tydra/scene-access.cc, line: 1752)
render-data (C++)   E  ConvertToRenderScene(env): VisitPrims fail (file: <...>/tinyusdz_repo/src/tydra/render-data.cc, line: 4897)

tellypresence commented 3 months ago

Another problem: it seems that render_scene.meshes[meshIdx].faceVertexIndices don't actually supply vertex indices but rather just the position in the array, i.e.

render_scene.meshes[meshIdx].faceVertexIndices[0] = 0
render_scene.meshes[meshIdx].faceVertexIndices[1] = 1
render_scene.meshes[meshIdx].faceVertexIndices[2] = 2
render_scene.meshes[meshIdx].faceVertexIndices[3] = 3
render_scene.meshes[meshIdx].faceVertexIndices[4] = 4
render_scene.meshes[meshIdx].faceVertexIndices[5] = 5

Instead of simple array position {0, 1, 2, 3, 4, 5, 6,...} would expect referring to meaningful vertex indices e.g. {0, 4, 6, 2, 3, 2, 6,...}

tellypresence commented 3 months ago

(moved comment to #130)

tellypresence commented 3 months ago

(moved comment to #130)

syoyo commented 3 months ago

@tellypresence Thanks! Will investigate face indices issue. Have you enabled triangulate in MeshConverterConfig?

tellypresence commented 3 months ago

Some more information about toy_car.usdz model; this loads OK on dev branch but fails on rendermesh-refactor branch.

Load logs for branch rendermesh-refactor (failure case):

tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 3341819
tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 741527
tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 3397576
tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 2138053
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 toy_car: IsXformablePrim1
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1981 local mat = ( (1.0, 0.0, 0.0, 0.0), (0.0, 1.0, 0.0, 0.0), (0.0, 0.0, 1.0, 0.0), (0.0, 0.0, 0.0, 1.0) )
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 Looks: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 toy_car_realistic_mat: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 file10: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 file11: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 file7: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 file8: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 file9: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 place2dTexture10: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 place2dTexture11: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 place2dTexture7: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 place2dTexture8: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 place2dTexture9: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 toy_car_realistic: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 toy_car_realistic_lod0: IsXformablePrim1
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1981 local mat = ( (1.0, 0.0, 0.0, 0.0), (0.0, 1.0, 0.0, 0.0), (0.0, 0.0, 1.0, 0.0), (0.0, 0.0, 0.0, 1.0) )
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:1975 toy_car_realistic_mat: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2001 Not xformable
tinyusdz_repo/src/tydra/render-data.cc:MeshVisitor:4615 Mesh: </toy_car/toy_car_realistic_lod0>
tinyusdz_repo/src/stage.cc:GetPrimAtPath:97 GetPrimAtPath : /toy_car/toy_car_realistic_lod0(input path: </toy_car/toy_car_realistic_lod0>)
tinyusdz_repo/src/stage.cc:GetPrimAtPath:117 clear cache.
tinyusdz_repo/src/tydra/shader-network.cc:GetBoundMaterial:431 search parent: /toy_car
tinyusdz_repo/src/tydra/render-data.cc:ConvertMesh:2815 uv primvar  with default_texcoords_primvar_name found.

Load logs for branch dev (success case):

tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 3341819
tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 741527
tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 3397576
tinyusdz_repo/src/tinyusdz.cc:LoadUSDZFromMemory:532 Image asset size: 2138053
tinyusdz_repo/src/tydra/render-data.cc:ConvertToRenderScene:2496 ConvertToRenderScene(): intro
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 toy_car: IsXformablePrim1
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2082 local mat = ( (1.0, 0.0, 0.0, 0.0), (0.0, 1.0, 0.0, 0.0), (0.0, 0.0, 1.0, 0.0), (0.0, 0.0, 0.0, 1.0) )
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 Looks: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 toy_car_realistic_mat: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 file10: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 file11: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 file7: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 file8: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 file9: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 place2dTexture10: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 place2dTexture11: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 place2dTexture7: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 place2dTexture8: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 place2dTexture9: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 toy_car_realistic: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 toy_car_realistic_lod0: IsXformablePrim1
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2082 local mat = ( (1.0, 0.0, 0.0, 0.0), (0.0, 1.0, 0.0, 0.0), (0.0, 0.0, 1.0, 0.0), (0.0, 0.0, 0.0, 1.0) )
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2075 toy_car_realistic_mat: IsXformablePrim0
tinyusdz_repo/src/tydra/scene-access.cc:BuildXformNodeFromStageRec:2102 Not xformable
tinyusdz_repo/src/tydra/render-data.cc:MeshVisitor:2392 Material: </toy_car/toy_car_realistic_lod0>
tinyusdz_repo/src/stage.cc:GetPrimAtPath:97 GetPrimAtPath : /toy_car/toy_car_realistic_lod0(input path: </toy_car/toy_car_realistic_lod0>)
tinyusdz_repo/src/stage.cc:GetPrimAtPath:117 clear cache.
tinyusdz_repo/src/tydra/shader-network.cc:operator():275 GPrim has no materialBinding.
tinyusdz_repo/src/tydra/shader-network.cc:FindBoundMaterial:327 search parent: /toy_car
tinyusdz_repo/src/stage.cc:GetPrimAtPath:97 GetPrimAtPath : /toy_car(input path: </toy_car>)
tinyusdz_repo/src/tydra/shader-network.cc:operator():275 GPrim has no materialBinding.
tinyusdz_repo/src/tydra/render-data.cc:ConvertMesh:1235 num_fvs = 76785
tinyusdz_repo/src/tydra/render-data.cc:MeshVisitor:2477 renderMaterialId = -1

syoyo commented 3 months ago

with https://github.com/syoyo/tinyusdz/commit/200130cfcf603ae11f414940a49223d5be55c443 and https://github.com/syoyo/tinyusdz/commit/4a5eef1511c26c1d8367225beb71508da5db3e91 , now at least tydra_to_renderscene does not produce conversion error for toy_car.usdz and fender_stratocaster.usdz

Fixing faceVertexIndices issue is the next step.

tellypresence commented 3 months ago

[Update] faces, normals, UVs, embedded textures working on recent rendermesh-refactor branch, reflected in updated screenshots below

Thank you for following up, confirm that with 2 latest commits, the 4 apple models which loaded OK on dev branch, also load successfully

(as expected, they have the same problem with faces, which will be addressed separately) ![Screenshot_20240418-091716]()

Model	Screenshots
Model	Solid	Wireframe
`toy_car.usdz`
`robot_walk_idle.usdz`
`fender_stratocaster.usdz`
`toy_drummer_idle.usdz`

tellypresence commented 3 months ago

The originally reported issue has been fixed, have raised separate issue for broken faces

syoyo / tinyusdz