Add `<canvas>` support for Replay

bruno-garcia commented 1 year ago

Problem Statement

The rrweb integration supports recording canvas. In Sentry we don't document this feature anymore because the product will not playback.

Solution Brainstorm

Add the ability to playback replays that involve canvas.

baruchoxman commented 1 year ago

We are a data platform, and a major part of our product is a Data ERD visualization on top of a canvas, so it's very important for us to capture how the users are using the ERD navigation. For more context, we're using a React wrapper on top of D3 Force Graph.

github-actions[bot] commented 1 year ago

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

drecali commented 1 year ago

My company works on image annotation platforms so canvas recording would be very helpful for us too.

yomansk8 commented 1 year ago

Hello,

In our product, we're using canvas in multiple place to display 3D interactive experience to our users. Sentry is a really great tool, and Session Replay is really helping us debugging/improving the user experience. But without canvas being captured, we are missing a lot of things, and it's harder to analyze.

Thanks again for the great work, and I hope this feature will come one day 🙏

rohaldb commented 1 year ago

Hi my company would like to use the screen recording feature but are blocked due to the lack of support for Canvas. Until its available i'm going to have to go with competitors

bruno-garcia commented 1 year ago

@rohaldb what solution do you currently use? Is it based on images or canvas command recording/playback?

rohaldb commented 1 year ago

Hi @bruno-garcia. We have a canvas rendered using konvajs that we would like to record. We use sentry for two reasons:

To view the interactions that caused errors (our canvas logic is quite complicated)
To sample user interactions so we can understand how they use our product

rohaldb commented 1 year ago

Oh sorry @bruno-garcia I think i misunderstood you. We are busy scoping out competitors and have not landed on one yet.

bruno-garcia commented 1 year ago

Latest update: We're running Replay on Sentry with rrweb 2.0 for a few weeks now. Feeling confident things are working pretty well. Soon we'll make this generally available and from there take steps to make canvas available

bruno-garcia commented 11 months ago

rrweb 2.0 version of Replay now merged into main https://github.com/getsentry/sentry-javascript/pull/8760 so we're unblocked to add canvas support. More updates to come.

goldengecko commented 11 months ago

Is there an ETA on this being available?

bruno-garcia commented 10 months ago

Is there an ETA on this being available?

Good timing! We're tacking this in Q3. We'll ping this thread to let folks know once we have a preview version available.

greengiraffe commented 10 months ago

We're tacking this in Q3.

@bruno-garcia Great news! Just to clarify, since we're already in Q4 2023, the Q3 was probably a typo, right?

lforst commented 10 months ago

We're tacking this in Q3.

@bruno-garcia Great news! Just to clarify, since we're already in Q4 2023, the Q3 was probably a typo, right?

@greengiraffe Hey, Q3 was indeed a typo we are looking at Q1 for this!

chambliss commented 10 months ago

Our product renders an embedding view in a canvas element (example), and being able to see the cluster quality of the embeddings in session replays would be very helpful. We are also relying on canvas for several charting elements in a more analysis-oriented view. Excited to try this out in Q1!

Arc-2023-11-14 at 12 13 36PM

Santas commented 9 months ago

Happy to test the preview when available.

bruno-garcia commented 9 months ago

Spoiler alert:

billyvg commented 9 months ago

We’re happy to announce that the early preview of canvas support has arrived for early adopters! Please upgrade your SDKs to version 7.90.0 to get experimental canvas support.

Please Read

We are opening this up the week before Christmas, know that there will be very limited (read: none) support during the week the Christmas (Dec 25 -> Jan 1). We'll be back in 2024 to help with any issues.
Ensure there is no PII in your canvas, we currently do NOT do PII scrubbing of the canvas
Recording in webgl contexts will force preserveDrawingBuffer to be true which can impact the performance of your canvas. We will be following-up with a method to manually snapshot your canvas inside of your repaint function so that we can keep preserveDrawingBuffer: false.
These APIs are very likely to change!

Installing Canvas for Replay

To get started, first make sure your organization is opted into the Early Adopters program:

sentry organization settings for early adopters

Next you'll need to add the package @sentry-internal/rrweb@2.6.0 to your workspace:

npm install @sentry-internal/rrweb

Finally, where you configure Sentry (e.g. call Sentry.init()) and Replay SDK:

import { getCanvasManager } from '@sentry-internal/rrweb';

Sentry.init({
  dsn: environment.SENTRY_DSN,
  ...
  integrations: [
    new Replay({
        ...
        _experiments: {
          canvas: {
            manager: getCanvasManager,
          },
        }
      },
    }),
  ]
});

You should be able to now see canvas in Replay!

GuskiS commented 8 months ago

Been getting some errors:

InvalidStateError: Failed to execute 'createImageBitmap' on 'Window': The ImageBitmap could not be allocated.
  at <anonymous>(../node_modules/@sentry-internal/rrweb/es/rrweb/packages/rrweb/src/record/observers/canvas/canvas-manager.js:134:17)
  at Array.forEach(<anonymous>)
  at takeCanvasSnapshots(../node_modules/@sentry-internal/rrweb/es/rrweb/packages/rrweb/src/record/observers/canvas/canvas-manager.js:123:25)

after which replay continues but canvas is no longer visible.

    "@sentry-internal/rrweb": "2.7.3",
    "@sentry/integrations": "7.92.0",
    "@sentry/react": "7.92.0",
    "@sentry/tracing": "7.92.0",
    "@sentry/vite-plugin": "2.10.2",

Otherwise seems to be running fine. Would love to have it bit smoother - user quickly moves mouse around but replay shows old canvas state.

lforst commented 8 months ago

@GuskiS Would you mind creating a separate issue with your setup and reproduction steps? Thanks!

GuskiS commented 8 months ago

@lforst Not sure how to reproduce this, this error is showing in sentry issues list from users when they interact with canvas.

lforst commented 8 months ago

@GuskiS Fair. It would still be cool if you could open a bug report with some more information so that we can effectively look and this and stay on-topic for this issue. Thank you!

mydea commented 8 months ago

We have also just released v7.94.1, which includes a new, better way to enable canvas capturing - can you give that a try and see if that possibly fixes things:

remove @sentry-internal/rrweb dependency (BTW you can also remove @sentry/tracing and just import that stuff directly from @sentry/react
Bump @sentry/react & @sentry/integrations to 7.94.1
Update your init config like this:

import * as Sentry from '@sentry/react'`;

Sentry.init({
  // ...
  integrations: [
    new Sentry.Replay(),
    new Sentry.ReplayCanvas(),
  ]
});

This should also ensure all versions etc. are in sync.

bruno-garcia commented 8 months ago

@GuskiS Would you mind creating a separate issue with your setup and reproduction steps? Thanks!

I raised it here, https://github.com/getsentry/sentry-javascript/issues/10271

tomasmozeris commented 6 months ago

Is there way to handle canvas with higher device pixel ratio, then width and height of it are doubled and scaled down, like in this tutorial: https://web.dev/articles/canvas-hidipi

Lms24 commented 6 months ago

Hey @tomasmozeris would you mind explaining in greater detail what currently doesn't work correctly with high DPI canvas? Do you have problems with recording or replaying the replay?

tomasmozeris commented 6 months ago

Hey @tomasmozeris would you mind explaining in greater detail what currently doesn't work correctly with high DPI canvas? Do you have problems with recording or replaying the replay?

I'm having issues while replaying canvas recording in sentry.io, use case for it would be if recorded threejs canvas on higher DPI monitor e.g: https://threejs.org/examples <canvas width="2262" height="2588" style="width: 1131px; height: 1294px; ;"></canvas> They set width and height attributes doubled and with style attribute shrinks it down. In sentry while replaying recording I see cut canvas view from top left corner and just half of original viewport size

billyvg commented 6 months ago

@tomasmozeris I've created a ticket for this here https://github.com/getsentry/sentry-javascript/issues/11203

billyvg commented 4 months ago

Closing out this ticket as canvas is now supported with replays. Canvas docs for reference. Please open a new ticket if you're encountering any bugs. Thanks!

getsentry / sentry-javascript