Would SVG screenshots work for playback?

[samreid]

We were investigating another way to support record and playback for research studies, one that is more robust than recording input events. I tried recording PNG screenshots, but it slowed down the performance quite a bit. What about creating "mostly svg" screenshots by getting the scenery div svg tree? Or even just saving the entire page HTML/css?

[samreid]

I tried the strategy at http://stackoverflow.com/questions/1750815/get-the-string-representation-of-a-dom-node and it worked pretty well aside from the embedding marks:

[samreid]

document.body.innerHTML works surprisingly well, but no WebGL and same embedding marks problem as above.

Update: studies could run with ?webgl=false

Update: canvas doesn't look captured either.

[samreid]

Accessing the innerHTML is very fast, but it something like 144,443 characters per frame for bending light. This is showing up as 145KB on my filesystem. It should be very compressible but that could slow things down. Perhaps we should re-investigate capturing small intermittent screenshots as a "ground truth" check against input event recording.

[jonathanolson]

I've used this approach for pixel-comparison unit tests, see Display.foreignObjectRasterization. There's some extra code that searches for Canvas elements, looks up their pixel content, and then replaces them with an image (with a data URL for the Canvas content).

For WebGL, this works, but requires the preserveDrawingBuffer Display flag (which triggers that on the WebGL Context, and can potentially slow down rendering performance).

Notably, this would be sharp (shows what is currently displayed), however may be platform-specific (Scenery might be using platform-specific features), and if there is a single screen-wide Canvas, then just simple screenshots would be smaller in size.

[samreid]

Linking to https://github.com/phetsims/phet-io/issues/1854 since @jonathanolson mentioned this strategy yesterday. Tagging @zepumph so he is aware.

[samreid]

Some other ideas for dealing with playback:

• https://github.com/rrweb-io/rrweb
• https://developer.mozilla.org/en-US/docs/Web/API/Screen_Capture_API

[samreid]

@matthew-blackman and @brent-phet and I discussed this problem today. Becoming aware that playbacks are no longer supported. We are unaware how important playback is for clients. Maybe @JacquiHayes or @kathy-phet can speak to the importance of record/playback. @samreid and @zepumph met with a client that needed something like this.

[brent-phet]

@kathy-phet mentioned a potential need to revisit Playback for a partner named TERC, possibly towards the end of 2024. I will bump in the priority here.