unoplatform / uno

Build Mobile, Desktop and WebAssembly apps with C# and XAML. Today. Open source and professionally supported.
https://platform.uno
Apache License 2.0
8.69k stars 700 forks source link

[WASM] Support for `RenderTargetBitmap` #10340

Open jeromelaban opened 1 year ago

jeromelaban commented 1 year ago

We'll need two parts for this to work (Since HTML does not allow to take screenshots easily).

The first is give the ability to an app to capture its content (given the user's consent) using something like this: Javascript provides this feature to capture the while screen:

const capture = async () => {
  const canvas = document.createElement("canvas");
  const context = canvas.getContext("2d");
  const video = document.createElement("video");

  try {
    const captureStream = await navigator.mediaDevices.getDisplayMedia();
    video.srcObject = captureStream;
    context.drawImage(video, 0, 0, window.width, window.height);
    const frame = canvas.toDataURL("image/png");
    captureStream.getTracks().forEach(track => track.stop());
    window.location.href = frame;
  } catch (err) {
    console.error("Error: " + err);
  }
};

capture();

What is not clear with this technique is that if we stop capturing, is there a second prompt showing up when capturing again. This would need some research. Also, given that the browser is asking for content, it may be best to place this feature behind a feature flag and explain what to do to enable it, and what the restrictions are.

The second part is more of an Uno.UITest change, which would require a new service that would take screenshots from an external process, during Runtime Tests on CI, but let's keep that one for later :)

amelield commented 1 year ago

@jeromelaban There is a second prompt if we capture two images and I am not sure if we can, or should, change that. Do full page screenshot works or do we want only the section with our structure, or both? Were do we want the image to be saved (or downloaded) and do we have documentation on how to put a feature behind a feature flag? Thank you!

jeromelaban commented 1 year ago

This would mean that we should be keeping the recording alive when it's started, so it does not prompt again. We can certainly keep a reference in our typescript layer.

amelield commented 1 year ago

Do we want to start the recording the first time we take a screenshot or at init. I think it would be more effective especially for someone who does not need it a lot. I am also working with the assumption that RenderTargetBitmap have to be accessible to all WASM and not only the Runtime test. Does this make sense?

jeromelaban commented 1 year ago

@amelield it should be started only on demand (it's probably adding some CPU load). Yes, it makes sense to provide that feature to all users (still behind the flag that we'll set to true in the samples app).

amelield commented 1 year ago

@jeromelaban I am sorry for the amount of question. I may have thought too much this week-end or not enough at the end of the week.

First of all, I just saw the section about [JSImportAttribute] and I think my code should use that for dotnet7. What is the test to make sure the version of dotnet used. Which file would be a good example of how to implement it differently for wasm dotnet6 and wasm dotnet 7? Where do public static partial class Imports for the [JSImportAttribute] would go in Uno?

Than my various questions:

Do the javascript script always have to be in a WasmScript section?

Should I touch Windows.Media.Capture partial class Screen Capture and the interface IMediaSource? I did not thought so since it seem disable for every platform. But I need to make sure.

Where do I put the activation (switch) in Sample and in which form?

Do I still need to be able to activate the code automatically in test or it is a problem for later? Where do we save the RawBitmap. Do we want the option to download the file too?

Thank you for you time and for any answer you can provide.