Three.js-based Animations

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

Add to Sage the ability to produce interactive 3D animations that can be explored by translating, rotating, and zooming as the animation progresses. Previously, an animation would need to be re-generated each time a different viewpoint was desired.

To that end, this ticket implements the following:

1. an .interactive() method on existing Animation objects that produces a new 3D graphics object containing all of the original frames of animation with additional animation metadata (keyframe assignments) attached that a supported viewer could use to depict the animation.

2. changes to the Three.js viewer to support keyframe animation of points, lines, texts, and surfaces when animation metadata is present including optional graphical controls allowing the user to play/pause, adjust playback position, control playback speed, and toggle looping.

3. support for saving a 3D graphics object directly to an HTML file that uses the Three.js viewer, bypassing the need to open it in a browser first and use the "Save as HTML" menu option.

4. support for saving an animation to an HTML file using features 1 and 3 listed above.

CC: @paulmasson @egourgoulhon @slel

Component: graphics

Keywords: threejs animation

Author: Joshua Campbell

Branch/Commit: 8af227d

Reviewer: Paul Masson

Issue created by migration from https://trac.sagemath.org/ticket/29194

[egourgoulhon]

comment:2

Looks very promising!

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:3

I opted not to implement persisting the animation state at this time.

Since the template is being used elsewhere, I went ahead and broke the animation javascript and CSS out into separate files in this initial iteration. The changes to the template file are now quite minimal, and I hope that makes porting future changes to it between Sage and other projects easier.

With the feature work complete, I'll move on to documentation and tests. One thing I've encountered so far is that the .. PLOT:: directive doesn't seem to work with Three.js plots. I'm curious if this is a problem in my installation of Sage or if it's a known issue. I noticed that the documentation for the Three.js viewer (src/doc/en/reference/plot3d/threejs.rst) embeds pre-generated HTML files so perhaps that's the direction I should go as well?

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:4

Joshua, I agree that it is better to separate the animation script as you have done. Have you verified that everything in the threejs folder gets copied during the build process?

The rendering of Three.js graphics requires a browser environment, unlike the PLOT directive. While it would be possible in principle to render WebGL at the command line using Node.js, that is a topic way outside of my interest as an HTML-oriented person. For that reason documentation is currently generated with Jmol as a static image. I am personally much more interested in the live HTML examples I put into the backend documentation and recommend you go in that direction as well. The only drawback is that the images won't appear in PDF versions of the documentation.

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:5

Replying to @paulmasson:

Joshua, I agree that it is better to separate the animation script as you have done. Have you verified that everything in the threejs folder gets copied during the build process?

I've verified that animation.js and animation.css are automatically copied after doing a full make build but not after a sage -b. The template behaves similarly. Do we want to try to ensure they get copied during a sage -b as well? I've been copying them all over manually in my own start script in order to avoid a full rebuild when none of the Python had changed.

The rendering of Three.js graphics requires a browser environment, unlike the PLOT directive. While it would be possible in principle to render WebGL at the command line using Node.js, that is a topic way outside of my interest as an HTML-oriented person. For that reason documentation is currently generated with Jmol as a static image. I am personally much more interested in the live HTML examples I put into the backend documentation and recommend you go in that direction as well. The only drawback is that the images won't appear in PDF versions of the documentation.

Ahh, that makes sense. I was forgetting about the PDF version. I will proceed with the separate HTML files for plots in the documentation then. Thank you :)

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:6

Replying to @jcamp0x2a:

Replying to @paulmasson:

Joshua, I agree that it is better to separate the animation script as you have done. Have you verified that everything in the threejs folder gets copied during the build process?

I've verified that animation.js and animation.css are automatically copied after doing a full make build but not after a sage -b. The template behaves similarly. Do we want to try to ensure they get copied during a sage -b as well? I've been copying them all over manually in my own start script in order to avoid a full rebuild when none of the Python had changed.

We won't have any control over the sage -b behavior since our files are not part of the core of Sage. make build or manually copying are the options.

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

Author: Joshua Campbell

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

Branch: u/gh-jcamp0x2a/29194-threejs-animation

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

Commit: dcd36c9

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:7

Hello, I hope this finds you well. I apologize for the delay in getting this ticket up for review, but I just didn't have a good feeling about the branch as it was.

After using it for a couple months, I've found the performance when using multiple animated variables to be quite wanting, as you'd expect from the exponential nature of adding new dimensions to the array of keyframes. Memory use and file size gets very large very quickly, especially when each frame itself is complex or if you've got several of them in the same notebook. Reducing the number of keyframes per variable helps, but having only a handful of frames makes noticing higher frequency stuff going on impossible. I consistently found myself using the interact command in the Jupyter notebook to control those extra variables instead.

I've pushed a new version of the branch for your review that drops support for multiple animated variables. A lot of the complexity of the old branch came from supporting them, so the new one is much simpler. It uses Three.js's built-in animation system now instead of a custom one, and it is invoked from the existing animation functionality users may already be familiar with: instead of calling an animation's gif or apng method, you call interactive. The name interactive was chosen instead of threejs to allow the possibility of other viewers supporting this in the future.

I've updated https://jcamp0x2a.github.io/threejs-animation-example/ with an example animated plot from the new branch.

Please let me know if you'd still prefer the older version. I believe the new one to be superior, but I'll try to keep up with and support either one going forward.

Thanks!

Joshua

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

1b6d2bf	Only animate when at least two frames present.
fd6f8a8	Fix misuse of SEEALSO block

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from dcd36c9 to fd6f8a8

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:9

Joshua, I’ll take a look at this once we’re into the next beta. Don’t want to make any major changes just before a release.

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:10

Replying to @paulmasson:

Joshua, I’ll take a look at this once we’re into the next beta. Don’t want to make any major changes just before a release.

Understood. I figured I missed the boat once I started seeing the '9.1rc' tags pop up in the git logs :)

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:12

Joshua, could you rebase this ticket off a current branch? That would save build time in reviewing. Thanks!

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fd6f8a8 to ee6d7bc

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

ee6d7bc

Add support for simple keyframe-based animation to Three.js viewer.

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:14

Hi Paul, I've rebased off of the latest in the develop branch. I verified that it still builds fine (hopefully the patch bot agrees) and quickly tested that the animation functionality is working as expected. I'll run a few more tests tonight or tomorrow as time permits to be sure, but it should be in a decent shape for review.

I imagine I'll want to rebase it again once your three.js update gets merged.

Thanks!

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:15

I've confirmed that the new version builds just fine and runs as expected. There is quite a lot of stuff changing here, so I'll need some time to read through all the code.

My first reaction is about the new bar across the top: although it's convenient for jumping to a frame, not sure if I like it yet. Maybe Eric can take a look and comment on it. The discrete controls would really be enough to control the animation.

Otherwise, it's looking great! Nice addition to Sage.

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:16

One other thought: shouldn't the decrease play back button be a reverse arrow? Or is this a Google standard icon?

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:17

Hi Paul,

I somewhat agree about the slider at the top, but I'm pondering future work where the animations aren't all or shouldn't necessarily all be discrete, in which case a way to navigate continuously would be helpful. For example: when just spinning an object about an axis there's no reason it shouldn't be transformed as continuously as the highest frame-rate the user/browser allows. Ideally, I'm imagining something like:

dodecahedron().rotateX(pi, duration=10).then().rotateY(-pi, duration=10)

The discrete frame-by-frame was a nice starting point, though, since that's what users currently do for 2D animations, and so only a small change is required: animate(frames).interactive() instead of, for example, animate(frames).apng().

Perhaps it's jumping the gun a bit, though it does also serve to show the animation progress. May come in handy for seamlessly looped animations to tell when one period ended and another began without a sharp cut as a visual cue.

---

Replying to @paulmasson:

One other thought: shouldn't the decrease play back button be a reverse arrow? Or is this a Google standard icon?

I used the hollow arrows for decrease speed instead of the reverse arrow since I didn't want the user to think it was a rewind button. I'm not particularly happy with the icons for the speed buttons, but I couldn't find anything in their icon set that seemed to fit better, and I'm not much of a graphics artist I fear.

Do you think chevrons instead of triangle arrows would convey the meaning better? > for slow down and >>> for speed up? That geometry doesn't seem too complicated so I'd probably be able to design it. Not up to Google's standards, I'm sure :)

---

I understand it's a good bit of code to look through, so no worries.

Thanks!

[egourgoulhon]

comment:18

Replying to @paulmasson:

My first reaction is about the new bar across the top: although it's convenient for jumping to a frame, not sure if I like it yet. Maybe Eric can take a look and comment on it. The discrete controls would really be enough to control the animation.

I took a look and I don't have a strong opinion on both options, so I leave it to you.

Otherwise, it's looking great! Nice addition to Sage.

Indeed! Thanks for this piece of work!

[mkoeppe]

comment:19

needs rebase

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ee6d7bc to 03c980f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

03c980f

Add support for simple keyframe-based animation to Three.js viewer.

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:21

One nitpick after reading through code: I don’t think you should change the documentation where you added “specifying an HTML file” since half of the following options aren’t supported. A bit confusing.

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:22

Looks mostly good to me, except for the issue raised about icons. I really think the slow down arrow should point to the left, but you're right that the rewind icon is misleading.

There are several of sets of left/right single chevrons already in the set of icons you're using: keyboard_arrow_left/right, navigation_before/after, chevron_left/right. Another choice would be single filled arrow_left/right.

Or perhaps even better switch_left/right, which shows a difference in direction immediately in the icons. For this choice I would have the filled arrow indicate the direction, which is opposite of the label. I think I like this last choice best.

[53959995-fd20-47f5-85e6-5e769b863d1f]

Reviewer: Paul Masson

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 03c980f to d48807c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

c5666a3	Change increase/decrease playback speed icons
d48807c	Revert a documentation change in Graphics3d.save

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:24

Hi Paul. I agree that that documentation change was confusing, so I've reverted it. I've also changed the icons to switch_left/right as requested, and I think they do indeed convey the meaning better. Here is an example of the new icons in action: https://jcamp0x2a.github.io/threejs-animation-example/simpler.html

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:25

Almost good to go: you didn't update the example with the new icons...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d48807c to 8af227d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

8af227d

Update animation example in Three.js documentation with new icons

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:27

Replying to @paulmasson:

Almost good to go: you didn't update the example with the new icons...

Ah! Thank you for catching that.

[53959995-fd20-47f5-85e6-5e769b863d1f]

comment:28

Cool! And you noticed that the examples for documentation should be generated using online=True. I'll add that to the template comment.

Thanks for your patience in the review process and slogging through multiple rebases. Nice work!

[slel]

comment:29

The ticket description reads

Tasks remaining before I'd be comfortable making a merge request:

If those tasks are now part of the ticket, please update the ticket description to

This ticket implements the following:

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

comment:30

Replying to @slel:

The ticket description reads

Tasks remaining before I'd be comfortable making a merge request:

If those tasks are now part of the ticket, please update the ticket description to

This ticket implements the following:

Apologies. The ticket description is indeed out-of-date. Many of the work items mentioned also became irrelevant following a change of course partway through. I will update the description to match the feature as currently implemented.

[21c3cf6e-6c50-4e8b-9831-5a03f9a599c1]

Description changed:

--- 
+++ 
@@ -1,37 +1,12 @@
-Moving conversation from #29192, where I mentioned that I've been working on adding support for three.js-based interactive 3D animated plots to Sage.
+Add to Sage the ability to produce interactive 3D animations that can be explored by translating, rotating, and zooming as the animation progresses. Previously, an animation would need to be re-generated each time a different viewpoint was desired.

-The current state of the work can be found at:
+To that end, this ticket implements the following:

-    https://gitlab.com/jcamp0x2a/sage/-/tree/threejs-animate
+1. an `.interactive()` method on existing `Animation` objects that produces a new 3D graphics object containing all of the original frames of animation with additional animation metadata (keyframe assignments) attached that a supported viewer could use to depict the animation.

-...and some examples of the plots generated so far and how the animation functionality is invoked:
+2. changes to the Three.js viewer to support keyframe animation of points, lines, texts, and surfaces when animation metadata is present including optional graphical controls allowing the user to play/pause, adjust playback position, control playback speed, and toggle looping.

-    https://jcamp0x2a.github.io/threejs-animation-example/
+3. support for saving a 3D graphics object directly to an HTML file that uses the Three.js viewer, bypassing the need to open it in a browser first and use the "Save as HTML" menu option.

-Tasks remaining before I'd be comfortable making a merge request:
+4. support for saving an animation to an HTML file using features 1 and 3 listed above.

-- Persist animation state across viewings. (maybe)
-
-- Better handling of variable name collisions. If both plot G1 and G2 use an animation variable `x`, rather than overwriting one of them when summing G1 + G2, rename one of them instead. `x`, `x′`, `x″`, etc.
-
-- Pause the render loop when no variable is being animated. (energy / battery life)
-
-- Partition scene objects into keyframe groups whose visibility can be toggled on/off in one assignment every frame vs. iterating over every scene object every frame.
-
-- Come up with several animated plots to embed in the documentation showing off a wide variety of plots being animated: simple shapes, parametric curves/surfaces, implicit surfaces, surfaces of revolution, scatter plots, vector fields, etc.
-
-- Tests!
-
-Possible future work:
-
-- Break animation javascript code off into a separate file that's only included when the plot contains animation. (smaller file size, less bandwidth, more modular)
-
-- Interpolate/blend/morph between keyframes in order to improve the appearance of the animations, and allow it to be disabled. Some ideas include:
-  - Fade keyframes in and out (via object opacity).
-  - Identify corresponding objects between frames and interpolate their transforms and/or geometries.
-  - Image-based blending: render the two keyframes you're between (for N variables, the min/max corners of the N-box) then blend them while rendering to a textured screen-size quad.
-
-- Performance improvements due to the # of objects animation may add to the scene: 
-  - Use Buffer Geometry instead of Geometry.
-  - Merge identical objects and use instanced rendering.
-  - Merge geometries that share a material and use range-restricted draws.

[slel]

comment:32

Thanks! Do open a follow-up ticket if some items were not covered here but would make sense.

[vbraun]

Changed branch from u/gh-jcamp0x2a/29194-threejs-animation to 8af227d