Sporadic blank frames (canvas/img)

[mifi]

Sometimes screenshot will take a screenshot before canvas/image has finished rendering.

Observations/testing:

• More easily reproduceable with high concurrency like 30
• Any other elements on the page will be screenshotted but not the canvas or img (type png or raw), which are sourced from the ffmpeg data: https://github.com/mifi/reactive-video/blob/65a19364f2b3e21fc2c813eb931ffd8140ff2a5d/packages/frontend/src/components/FFmpegVideo.js#L39
  • I tried a scene with one Image, one FFmpegVideo and some text - only the Image and text got rendered
• From testing, I have confirmed that ffmpeg is not returning white frames
  • There are never any white frames returned from the canvas's ctx.getImageData()
• when adding a border to the canvas or image, the border does not show in the screenshot when the issue is triggered, so the html element is really not rendered yet at the time of screeshot
• also converting canvas to canvas.dataURL() and setting on src on img yields the same sporadic issue
• multiple setTimeout(..., 0) and multiple requestAnimationFrame does not help. (or multiple awaitDomRenderSettled)
• png seems to produce more frequently than raw (maybe due to higher CPU usage)
• captureType screenshot can reproduce it but screencast seems to reproduce even more often
• Tends to happen at the first frame after puppeteer boots up (on the first frame of some random part when multiple parts), maybe due to high CPU

Workarounds

• setTimeout(500) or page.waitForNetworkIdle (which I think also waits for 500ms) will workaround this issue but it's a hack because it's not real synchronisation and may in theory fail some time.
• Render the first frame of each part twice and check if equal, or just always render first frame of each part twice

Alternatives

Doesn't work

const rect = imgRef.current.getBoundingClientRect();
if (!rect.width || !rect.height) throw new Error('wh');

await new Promise((resolve, reject) => {
  const observer = new IntersectionObserver((entries) => {
    if (entries[0].isIntersecting) {
      // el is visible
      resolve();
    } else {
      // el is not visible
      reject(new Error('invis'));
    }
  });
  observer.observe(imgRef.current);
})

await new Promise((resolve, reject) => {
  let timeout2;
  const timeout = setTimeout(() => {
    clearTimeout(timeout2);
    reject(new Error('Image never completed'));
  }, 10000);

  function checkComplete() {
    if (imgRef.current.complete && imgRef.current.naturalWidth > ) {
      clearTimeout(timeout);
      resolve();
      return;
    }

    timeout2 = setTimeout(checkComplete, 100);
  }

  checkComplete();
});

HeadlessExperimental frame control

• https://docs.google.com/document/d/1PppegrpXhOzKKAuNlP6XOEnviXFGUiX2hop00Cxcv4o/edit#
• https://groups.google.com/a/chromium.org/g/headless-dev/c/S5CoLs46AiE?pli=1

Not tried

• callback from ReactDOM.render(element, container[, callback]) https://reactjs.org/docs/react-dom.html#render
• https://pptr.dev/#?product=Puppeteer&version=v5.2.1&show=api-elementhandleboundingbox
• https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagewaitforselectorselector-options https://github.com/puppeteer/puppeteer/blob/1de9906260269491c9011fcab33a58e280603fd3/lib/DOMWorld.js#L505

Links

• https://stackoverflow.com/questions/15875128/is-there-element-rendered-event
• https://stackoverflow.com/questions/26556436/react-after-render-code

Possibly related

• https://github.com/puppeteer/puppeteer/issues/5352

[mifi]

Update: The img bug was due to broken ffmpeg node.js stream code. After fixing that bug, I have not seen this bug again when using images instead of canvas. Because canvas is no longer used by default I'm closing this.