phetsims / scenery

Scenery is an HTML5 scene graph.
MIT License
54 stars 12 forks source link

Would SVG screenshots work for playback? #561

Open samreid opened 8 years ago

samreid commented 8 years ago

We were investigating another way to support record and playback for research studies, one that is more robust than recording input events. I tried recording PNG screenshots, but it slowed down the performance quite a bit. What about creating "mostly svg" screenshots by getting the scenery div svg tree? Or even just saving the entire page HTML/css?

samreid commented 8 years ago

I tried the strategy at http://stackoverflow.com/questions/1750815/get-the-string-representation-of-a-dom-node and it worked pretty well aside from the embedding marks:

image

samreid commented 8 years ago

document.body.innerHTML works surprisingly well, but no WebGL and same embedding marks problem as above.

Update: studies could run with ?webgl=false

Update: canvas doesn't look captured either.

samreid commented 8 years ago

Accessing the innerHTML is very fast, but it something like 144,443 characters per frame for bending light. This is showing up as 145KB on my filesystem. It should be very compressible but that could slow things down. Perhaps we should re-investigate capturing small intermittent screenshots as a "ground truth" check against input event recording.

jonathanolson commented 8 years ago

I've used this approach for pixel-comparison unit tests, see Display.foreignObjectRasterization. There's some extra code that searches for Canvas elements, looks up their pixel content, and then replaces them with an image (with a data URL for the Canvas content).

For WebGL, this works, but requires the preserveDrawingBuffer Display flag (which triggers that on the WebGL Context, and can potentially slow down rendering performance).

Notably, this would be sharp (shows what is currently displayed), however may be platform-specific (Scenery might be using platform-specific features), and if there is a single screen-wide Canvas, then just simple screenshots would be smaller in size.

samreid commented 2 years ago

Linking to https://github.com/phetsims/phet-io/issues/1854 since @jonathanolson mentioned this strategy yesterday. Tagging @zepumph so he is aware.

samreid commented 1 year ago

Some other ideas for dealing with playback:

samreid commented 1 year ago

@matthew-blackman and @brent-phet and I discussed this problem today. Becoming aware that playbacks are no longer supported. We are unaware how important playback is for clients. Maybe @JacquiHayes or @kathy-phet can speak to the importance of record/playback. @samreid and @zepumph met with a client that needed something like this.

brent-phet commented 10 months ago

@kathy-phet mentioned a potential need to revisit Playback for a partner named TERC, possibly towards the end of 2024. I will bump in the priority here.