Canvas instrumentation (recording)

[vaind]

Use case

I'm looking into instrumenting canvas calls to be able to record all paint operations for future replay outside Flutter.

Proposal

I've experimented with this and it seems I can achieve much of the goal by providing a custom PaintingContext that provides a proxy Canvas which records & forwards all calls to the standard canvas.

There's a single place where a new PaintingContext is constructed: https://github.com/flutter/flutter/blob/50a23d962c887154e56d47ffa0bae6efa2b9e21b/packages/flutter/lib/src/rendering/object.dart#L169 Providing a way to provide a factory for this place would enable providing a custom PaintingContext as well as a custom Canvas.

This could be achieved, for example, by exposing an overridable static factory callback

  static PaintingContext Function(ContainerLayer, Rect) repaintCompositedChildFactory = PaintingContext.new;

I can provide a PR, just wanted to discuss first which way to do this would be most desirable.

---

Additionally, to achieve the overall goal, that is to record canvas paint operations, there two non-trivial objects that currently cannot be reflected (and recorded), i.e. ui.Path and ui.Paragraph. I'm not yet sure what the best approach would be. They're constructed with their respective builders so one approach would be to hook into the builders. Another yet could be to provide a way to serialize those objects on demand.

[marandaneto]

@goderbauer would you be fine with the suggested changes, can we go ahead with a Draft PR?

[tvolkert]

We'd probably want to (1) only allow this in debug and profile builds, and (2) turn it on via an rpc command to the observatory (or maybe implement it as a "take a snapshot" command where you capture one frame).

[bruno-garcia]

Our goal is to record things in production builds. Aiming to have an overhead low enough that's not noticible of course. So debug/profile build only wouldn't really help us. Any chance we can have this available also in prod builds? If unused, shouldn't really affect performance, correct?

[goderbauer]

Could you shine a little light on what your use case for this is? Why do you want to record this in production? What do you want to do with the information? For these kind of discussions it's usually more fruitful to start with a problem instead of suggesting a solution.

[bruno-garcia]

Our goal is to build a Session Replay feature for Flutter.

[bruno-garcia]

@goderbauer @tvolkert any major objections we could try to work around? We'd love to build good developer tools for Flutter devs

[The-RootCause]

Any update on this feature. This is very important but still considered as P3

[bruno-garcia]

We didn't give up on building Replay for Flutter. We took another route. Updates coming soon :)

[Jesse-Abruzzo]

any updates?

[jtavio]

any update on this issue?