Feature request: Load entire existing three.js scene into theatre sheet

[hybridherbst]

This would serve two purposes:

• makes it easy to use theatre for reviewing animated files that have been created elsewhere (e.g. Babylon Sandbox has a pretty unusable animation sheet, theatre could improve upon that)
• makes it easy to stresstest with real-world files and hundreds/thousands of keyframes.

Also curious about pointers to dabble with the above if this is out of scope :)

[donmccurdy]

There's a fair bit of code in the theatre/r3f package that generates theatre 'editable' types (i.e., the glue code setting up theatre scene hierarchy UI and property controls, which you'd need to start animating a scene) from R3F objects. It could be applied to vanilla three.js objects with syntax like <e.primitive object={object} editableType={object.type} /> in most cases, by traversing the scene, but is React-dependent.

I wonder if you don't strictly want to load a scene so much as an existing animation clip, though? Probably also with the goal of making changes and exporting the result back to a THREE.AnimationClip or a glTF file? I don't know enough about the internal representation of keyframes in theatre to guess how well this would work, though. glTF allows STEP, LINEAR, and CUBICSPLINE interpolation modes, with the latter being more constrained (for runtime purposes) than Blender or theatre.js' more flexible Bezier handles. There's a good thread on the differences here: https://blender.stackexchange.com/questions/269337/what-b%c3%a9zier-animations-can-be-safely-exported-to-gltf

Another idea would be to write some code that syncs theatre.js keyframes to/from a glTF-Transform AnimationSampler object — then hooking up animation support in gltf-transform/view, which I unfortunately haven't gotten around to yet. With that approach you'd have a pretty good workflow to read a glTF, do some keyframe animation, and export back to glTF losslessly. But probably a bit outside the scope of theatre.js itself. :)

[jo-chemla]

I've been trying to understand whether it is possible or not to import/export glTF animations interpolation (step, linear, linear spherical for rotations, cubic spline) to/from a theatrejs project for easier IO/VFX/compositing within softwares that can read/write gltf like Blender/UE via their respective addons.

Thanks for these resources regarding (blender/theatre) bezier <-> (gltf) cubic-spline curve conversion and this 1/3-2/3 rule. It could make sense to enforce these constraints on the user-chosen handles so that the animation can seamlessly be transported from one framework to the other (both being bezier) while the intermediate gltf container would use cubic-spline curves. These constraints could be enforced either when the user sets the handles - force handle x to 1/3 and have mouse cursor control only handle y - or simpler correcting all the handles at once with the user defined direction and editing their timestamps to 1/3-2/3.

Would there be another way to transport these bezier curves from/theatre/blender?

• write a blender python bpy script which would import a theatrejs json project (sad to not use a standard format containing these animations and interpolations) OpenUSD seem to support Bezier/Bspline/catmullRom, would it make sense to make a theatrejs exporter only for these animations as bezier curve? Or is gltf at some point going to support bezier within the spec?
• bad hack interpolate curve values between keyframes for every frame, and export not only the keyframes but one value per keyframed prop for every frame.

[donmccurdy]

bad hack interpolate curve values between keyframes for every frame, and export not only the keyframes but one value per keyframed prop for every frame.

In practice, this is what Blender usually does when exporting to glTF. Its own keyframe editor is far more flexible than what realtime engines are likely to support. In theory a DCC tool or an artist could constrain themselves to only the cubic spline interpolations the realtime specs require, but I imagine that's a tough sell.

Linearly sampling every keyframe is pretty fast, and the result can de thinned to not contain much redundant data, or compressed with Meshopt in glTF. That's what I'd generally do for last-mile delivery to runtime, and it might be the place to start here. But if you want to bring the animation into Blender and then edit it further there, the sampled keyframes are much harder to work with.

[jo-chemla]

Thanks for that feedback, indeed just realized that Blender gltf exporter evaluates keyframed props at every frame of the animation upon export - so theatrejs gltf-export could indeed do the same. Animation would indeed be defined in one tool and could be imported as gltf for pixel-perfect compositing. As far as I understand, the thinning would be useful only for Step keyframe types - where prop is constant - but indeed reducing redundancy and compressing would be helpful.

It's true, constraining 3d-artists to cubic-spline is hard to understand, pretty sad that the gltf standard does not allow for Cubic spline interpolation. Since you really know the spec well, do you have any idea whether expanding animation support to cubic-spline has been discussed?

Having theatrejs export anim to a filetype which supports bezier would be a really useful feature for such use-cases. It could make sense to use

• OpenUSD as a container for these animations since it seems to support these splines. But it seems that the blender exporter/importer does not register standard keyframes - or at least I cannot see them upon import?
• Alembic abc, same thing blender exporter seem to interpolate curve values at every frame.

[Edit] Found this thread of the gltf-spec also mentioning the 1/3-2/3 handles rule. Could maybe bezier-spline support be added as a EXT_* gltf extension - so not part of the core spec, but something useful for transporting such animations for softwares/users that could benefit from exporting the curve paths defined via their keyframes rather than for every frame?

[jo-chemla]

PS just saw you moved the @gltf-transform/view into the gltf-transform monorepo (packages/view and roadmap).

Could this view utility be used as well in place of gltfjsx to:

• convert gltf to r3f components on the fly,
• or to theatrejs r3f components - or threejs/theatre editable scene like the OP aims for? Thanks!

[donmccurdy]

As far as I understand, the thinning would be useful only for Step keyframe types - where prop is constant - but indeed reducing redundancy and compressing would be helpful.

Thinning is effective for both Step and Linear interpolation. In practice, Linear is the critical one – see below.

... pretty sad that the gltf standard does not allow for Cubic spline interpolation. Since you really know the spec well, do you have any idea whether expanding animation support to cubic-spline has been discussed?

I have no objection to seeing additional interpolation types in the glTF spec, perhaps that would be a good thing. However, bringing original keyframes from a DCC tool has other challenges:

• Blender supports per-keyframe selection of interpolation type. glTF and three.js require the entire keyframe track/channel (all keyframes affecting a specific property of a specific object for an animation) to have the same interpolation type. I'm not sure about other realtime 3D engines, but per-keyframe interpolation would be slower to execute.
• Blender's constraints will always require resampling keyframes.
• Blender rotations are usually keyframed in Euler coordinates, which must be converted to quaternions for realtime things like glTF, three.js, and Unity. From the Unity docs:

Unity’s default behavior is to resample [Euler] animations as Quaternion values and generate a new Quaternion keyframe for every frame in the animation.

For all of these reasons, it's hard to say what would even be required of glTF, for Blender to never resample keyframes on export. OpenUSD is a very flexible format (which creates challenges on the import side...) but even there, I'm not sure whether OpenUSD <-> Blender is lossless.

When resampling a cubicspline or bezier curve at every keyframe, it's standard to use linear interpolation for the resampled channel, approximating the original curve as a piecewise-linear function. Resampling (as described above) and compression are then often very effective, even though these were originally non-linear curves.

tl;dr — If you do not absolutely need the original keyframes 1:1, then creating an optimized, runtime-ready representation of the animation curves (with resampling and compression) is a considerably easier problem than all of the above.

---

Could this view utility be used as well in place of gltfjsx to ...

I'll comment on the gltfjsx thread below – it probably depends what is meant by "r3f components on the fly".

• https://github.com/pmndrs/gltfjsx/issues/255

[jo-chemla]

Thanks for these insights, really useful to understand the implications of such decisions and to evaluate what would be the best route in the described use-case. Also thanks, duly noted the tldr that building custom code for easier IO export/import of these keyframe setups would be best in this case - at least while there is not a more generic way to handle keyframes IO between DCC tools, via gltf or usd. I'm very interested to follow the discussion whether such discussions make it to the gltf spec somehow.

• Sorry for the thread hijack Herbst!