Off-screen rendering with webview2

[ajaymonga]

Hi Is it possible to get webview2 to render into a shared memory region like CEF : https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-header-off-screen-rendering

In my application I use 2 process architecture where main process does not have network access and second process uses CEF to render webcontent into a shared memory region from where main app can read the pixels. I am wondering if I can achieve this using webview2

Thanks

AB#28491736

[MichaelPfob]

1. WinForms but flexible
2. WebKit based into bitmap buffer
3. Not directly but possible; getting a bitmap as convenience functionality would be great

UseCase: Rendering SVG images into raster images

[thomasreiser]

@nishitha-burman @champnic any news on that topic you could share with us? Do you need further feedback from us in order the finalize this feature?

[etkramer]

Also posting here to express interest in the feature!

1. Win32/WinForms
2. Direct3D 12
3. Absolutely

[microdee]

expressing interest too:

1. Win32 (native no .NET)
2. D3D12 / D3D11 / Vulkan
3. That would be the best.

[AshleyScirra]

I think this issue is relevant to being able to show the Steam Overlay over WebView2. So far I've found it extremely difficult to do this, as covered in this blog post I wrote about it. In short the Steam Overlay does not understand the multi-process architecture of WebView2. The app can work around this by creating its own D3D11 swapchain and layering it over WebView2 with DirectComposition. That doesn't work with transparency though as the Steam Overlay expects to render over an opaque background, and the alpha blending looks all wrong.

AFAICT the Steam Overlay renders itself in the call to Present() and so the app itself has almost no control over how it's actually rendered, which severely limits any ability to work around these problems. However I think if WebView2 could render itself to a texture or a swapchain, then I think it should work correctly. Then the app could create its own D3D11 swapchain, render WebView2 in to it, and then call Present(), at which point the Steam Overlay draws over the existing WebView2 content.

To answer the prior questions for our specific case:

1. Win32, C++
2. Our app only shows WebView2, but also needs to show the Steam Overlay.
3. Rendering to ID3D11Texture2D would work as the app could paste that in to the swapchain and then call Present(). I'm not sure if IDXGISurface would support this case, but I suspect it probably would, so long as it belongs to the app process and not the WebView2 process so Steam can draw over it.

[gganjang]

1. Win32
2. Vulkan
3. ID3D11Texture2D preferred

[mdrejhon]

More info about TestUFO use case; I posted before, but expanding:

1 -- VRR support consideration:

I desparately want to make a standalone TestUFO executable wrapper around TestUFO motion test, and play to a variable refresh rate Direct3D. (So make sure requestAnimationFrame() refresh cycle callback in the HTML renderer able to tick-tock properly to a frame rate unsynchronized to the display's max-Hz refresh rate). In other words, I want to override the offscreen browser's "assumed refresh rate", and make it tick tock to the frame rate of the Direct3D presentation instead. This will allow me to support VRR in a web browser engine, simply by building an executable Direct3D wrapper around TestUFO (aka a custom browser that uses Edge engine, just to support VRR).

2 -- HDR support consideration:

Microsoft Edge now also has a hidden browser flag that enables HDR ( edge://flags/#enable-experimental-web-platform-features supports enabling 48-bit HDR framebuffers, as does Chrome too). It even supports whiter than white, in my upcoming browser-based TestUFO 2.0 HDR, optionally producing whites for scientific motion tests (I'm in over 30 peer reviewed papers).

The top part of the screen is hex #FFFFFF, while that white square is rec2020(5,5,5) which can peak at pretty bright on the new 1000nit HDR gaming displays, used for showing nice highlights (in games it's often only on 1% of pixels; good neons, good sunglints on metal, etc)

FFFFFF (surrounding background) versus color(rec2020 5 5 5):

This HDR version is coming out for CES 2024, but I want to support VRR sometime later in 2024 through one of many possible workflows (including, of course, possibly this one). Note, this website is often used by gamers, manufacturers, researchers, to test their displays, including OLEDs, LCDs, gaming displays, etc.

So ideally -- when supporting browser rendering, please also support the existing HDR Direct3D, with an unbroken colorspace-conversion chain (untouched pixels as possible). I am researching other workflows, but I just wanted to mention this additional consideration -- offscreen HDR Edge rendering to a HDR video game.

[fredemmott]

1. C++ Win32, mix of raw Win32 APIs, WinUI3, C++/WinRT, OpenXR, Oculus API, and OpenVR. None of those XR APIs (or even Windows Holographic) have any DirectComposition support
2. Direct2D and Direct3D11; interacts with other programs, so using existing D3D12 and Vulkan interop
3. Yes, strongly preferred. Being able to display animations/videos efficiently is highly desirable, and overhead of main RAM buffers is too high.

[fredemmott]

So, I finally ended up taking the Windows::Graphics::Capture approach. Using full C++/WinRT but still wanting render-to-texture is probably a little rare, but in case anyone else wants the modern stuff including co_await rather than COM + std::future:

Firstly, you want these docs: https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/winrt/microsoft_web_webview2_core/ - whenever you google, you'll get the classic COM C++/Win32 docs instead, or the UWP docs. This is one of the few places in the WinRT world where the UWP API differs significantly.

With that done, first, there's some objects you'll want to keep track of:

  unique_hwnd mBrowserWindow;

  winrt::Windows::UI::Composition::Compositor mCompositor {nullptr};
  winrt::Windows::UI::Composition::ContainerVisual mRootVisual {nullptr};
  winrt::Windows::UI::Composition::ContainerVisual mWebViewVisual {nullptr};

  winrt::Microsoft::Web::WebView2::Core::CoreWebView2Environment mEnvironment {
    nullptr};
  winrt::Microsoft::Web::WebView2::Core::CoreWebView2CompositionController
    mController {nullptr};
  winrt::Microsoft::Web::WebView2::Core::CoreWebView2 mWebView {nullptr};

You must explicitly initialize these to nullptr, otherwise they will be default-constructed as real instances, probably on a different thread to the one you want.

Register a window class:

  WNDCLASSW windowClass {
    .style = CS_HREDRAW | CS_VREDRAW,
    .lpfnWndProc = &WebView2PageSource::WindowProc,
    .hInstance = GetModuleHandle(nullptr),
    .lpszClassName = WindowClassName,
  };
  ::RegisterClassW(&windowClass);

... and from here on, it needs to be in the Windows::Graphics::Capture DispatcherQueue's thread:

Create an invisible window that is a child of HWND_MESSAGE instead of HWND_DESKTOP:

  mBrowserWindow = unique_hwnd {CreateWindowExW(
    0,
    WindowClassName,
    L"YOUR APP HERE WebView2 Host",
    0,
    CW_DEFAULT,
    CW_DEFAULT,
    0,
    0,
    HWND_MESSAGE,
    NULL,
    GetModuleHandle(nullptr),
    nullptr)};

Set up Windows::UI::Composition:

  mCompositor = {};
  mRootVisual = mCompositor.CreateContainerVisual();
  mRootVisual.Size({WIDTH, HEIGHT});
  mRootVisual.IsVisible(true);

  mWebViewVisual = mCompositor.CreateContainerVisual();
  mRootVisual.Children().InsertAtTop(mWebViewVisual);
  mWebViewVisual.RelativeSizeAdjustment({1, 1});

Set up webview 2:

  using namespace winrt::Microsoft::Web::WebView2::Core;
  CoreWebView2EnvironmentOptions options;

  const auto userData = Filesystem::GetLocalAppDataDirectory() / "WebView2";
  std::filesystem::create_directories(userData);

  const auto windowRef
    = CoreWebView2ControllerWindowReference::CreateFromWindowHandle(
      reinterpret_cast<uint64_t>(mBrowserWindow.get()));

  mEnvironment = co_await CoreWebView2Environment::CreateWithOptionsAsync(
    {}, userData.wstring(), {});
  mController
    = co_await mEnvironment.CreateCoreWebView2CompositionControllerAsync(
      windowRef);
  mWebView = mController.CoreWebView2();

  auto settings = mWebView.Settings();
  const auto userAgent = std::format(
    L"{} YOUR_APP_HERE/{}.{}.{}.{}",
    std::wstring_view {settings.UserAgent()},
    Version::Major,
    Version::Minor,
    Version::Patch,
    Version::Build);
  settings.UserAgent(userAgent);

  mController.BoundsMode(CoreWebView2BoundsMode::UseRawPixels);
  mController.RasterizationScale(1.0);
  mController.ShouldDetectMonitorScaleChanges(false);
  mController.Bounds({
    0,
    0,
    mSize.Width<float>(),
    mSize.Height<float>(),
  });

  mController.RootVisualTarget(mWebViewVisual);
  mController.IsVisible(true);

  mWebView.Navigate(L"https://www.testufo.com");

... and finally, create the Windows::Graphics::Capture::GraphicsCaptureItem:

 winrt::Windows::Graphics::Capture::GraphicsCaptureItem::
    CreateFromVisual(mRootVisual);

Then use WGC as normal.

Edit: This is just isolating and WinRT-ifying @jnschulze's Flutter work with COM and std::future over on https://github.com/MicrosoftEdge/WebView2Feedback/issues/20#issuecomment-1030650428 ; they figured out this approach :)

[fredemmott]

For off-screen rendering in particular, you probably want to disable v-sync, and to cap the framerate:

  const auto edgeArgs
    = std::format(L"--disable-gpu-vsync --max-gum-fps={}", FramesPerSecond);
  CoreWebView2EnvironmentOptions options;
  options.AdditionalBrowserArguments(edgeArgs);

  mEnvironment = co_await CoreWebView2Environment::CreateWithOptionsAsync(
    {}, userData.wstring(), options)

[mdrejhon]

Thanks for finding a solution to this! I need to try this.

You even found a solution for the independent-framerate capability, which is what I needed.

The only missing piece is whether there is a 48-bit HDR Edge WebView framebuffer — since I will need to support HDR. Edge already supports HDR graphics (Good browser test page for HDR: https://gregbenzphotography.com/hdr/ …).

Both Edge and Direct3D supports HDR, but will a WebView (yet) support HDR? And if Windows::Graphics::Capture supports copying other color depths other than 24-bit RGB (HDR 48-bit, e.g. rec2100-pq or rec2100-hlg or display-p3)…

I can make do without HDR, but it may be a good time for Microsoft to make sure their API-chain is at least HDR-ready, so there’s no color-decimation in the path.

[fredemmott]

WGC definitely supports HDR via DirectXPixelFormat::R16G16B16A16Float for real windows; I've not tried for WebView2 as for my use case I always want SDR, even on HDR systems.

[fredemmott]

@mdrejhon also, thanks, testufo was my go-to test to make sure I had this plumbed up correctly :) As you mentioned, VRR doesn't work, but it was still reaching ~ 238hz when it reaches 240hz on a v-synced chrome.

[softworkz]

@fredemmott - Great stuff! Are you able to achieve transparency this way?

[fredemmott]

@fredemmott - Great stuff! Are you able to achieve transparency this way?

Yep:

  mController.DefaultBackgroundColor(winrt::Windows::UI::Colors::Transparent());

Then when I use devtools to add some border-radius:

[softworkz]

You mean the border radius of the buttons? Is this real transparency or just blending against the background visual of the window? Does it work when there's an image or other content behind?

[fredemmott]

You mean the border radius of the buttons? Is this real transparency or just blending against the background visual of the window? Does it work when there's an image or other content behind?

Real RGBA data in the D3D11 texture. That's not a window background, it's 3D scene with the webview2 rendered as a quad - moving view so there's an object at the top left corner here:

[softworkz]

Thanks for the clarification - it wasn't even clear ot me which part is the webview content 😆

Anyway I just mixed this up with Microsoft.Composition, but you are actually using Windows.Composition where WebView2 transparency is doing just fine. Thanks again.

[mdrejhon]

@mdrejhon also, thanks, testufo was my go-to test to make sure I had this plumbed up correctly :) As you mentioned, VRR doesn't work, but it was still reaching ~ 238hz when it reaches 240hz on a v-synced chrome.

You can get perfect 240fps 240Hz if you exit Task Mananger and your background RGB controller software, and not run other browser tabs. TestUFO has been tested up to 1000Hz already.

Now…

VRR TestUFO may be achieved this way

• Direct3D Executable app
• Capped frame rate in WebView2 using your info
• Windowed GSYNC enabled in Control Panel
• VRR frame rate configured by user (command line or app settings screen)
• Direct3D frame rate sync’d to WebView2 frame rate

Now while I do C# with a bit of dabble in C++…. I have never done Direct3D programming from scratch (only modifications) — does anyone have a Hello World web browser demo for Direct3D that is MIT/Apache or something reasonably permissive?

Does not have to support all this nor VRR, just a dummy working Direct3D WebView2 app I can modify to suit needs. A code skeleton to work from, hardcoded to a URL or offline HTML file. Preferably WebView2 texture displaying 1:1 mapped to full screen, for simplicity’s sake.

If nothing open source already exists, I may be willing to pay for this small starter code skeleton (www.blurbusters.com/about/contact) if there isn’t already an open source harness or windows sample that I can modify…

[fredemmott]

@mdrejhon also, thanks, testufo was my go-to test to make sure I had this plumbed up correctly :) As you mentioned, VRR doesn't work, but it was still reaching ~ 238hz when it reaches 240hz on a v-synced chrome.

You can get perfect 240fps 240Hz if you exit Task Mananger and your background RGB controller software, and not run other browser tabs. TestUFO has been tested up to 1000Hz already.

Now…

VRR TestUFO may be achieved this way

• Direct3D Executable app
• Capped frame rate in WebView2 using your info
• Windowed GSYNC enabled in Control Panel
• VRR frame rate configured by user (command line or app settings screen)
• Direct3D frame rate sync’d to WebView2 frame rate

Now while I do C# with a bit of dabble in C++…. I have never done Direct3D programming from scratch (only modifications) — does anyone have a Hello World web browser demo for Direct3D that is MIT/Apache or something reasonably permissive?

Does not have to support all this nor VRR, just a dummy working Direct3D WebView2 app I can modify to suit needs. A code skeleton to work from, hardcoded to a URL or offline HTML file. Preferably WebView2 texture displaying 1:1 mapped to full screen, for simplicity’s sake.

If nothing open source already exists, I may be willing to pay for this small starter code skeleton (www.blurbusters.com/about/contact) if there isn’t already an open source harness or windows sample that I can modify…

Sorry, I wasn’t clear - I have no trouble with VRR 240 in chrome; i

@mdrejhon also, thanks, testufo was my go-to test to make sure I had this plumbed up correctly :) As you mentioned, VRR doesn't work, but it was still reaching ~ 238hz when it reaches 240hz on a v-synced chrome.

You can get perfect 240fps 240Hz if you exit Task Mananger and your background RGB controller software, and not run other browser tabs. TestUFO has been tested up to 1000Hz already

sorry for being unclear: I have no trouble with 240fps with gsync in chrome; that is not translating to the same with WGC+webview2

[mdrejhon]

sorry for being unclear: I have no trouble with 240fps with gsync in chrome; that is not translating to the same with WGC+webview2

Aha. Browser frame rate debugging can be done using www.testufo.com/animation-time-graph

Also, if you know where I can get a code sample of this, let me know…

[fredemmott]

Also, if you know where I can get a code sample of this, let me know…

Not isolated, and doesn't meet your license requirements (it's GPLv2 with an exception for the Windows App SDK which I'm not convinced is GPL-compatible given it can only be built by MS employees), but for others or in case it's useful anyway, it's https://github.com/OpenKneeboard/OpenKneeboard/blob/master/src/app/app-common/PageSource/WGCPageSource.cpp + https://github.com/OpenKneeboard/OpenKneeboard/blob/master/src/app/app-common/PageSource/WebView2PageSource.cpp ; structured like that as I also have Window Capture as a content source, among others.

[mdrejhon]

Ah, the TestUFO requires something a bit more permissive (MIT/Apache) since I'd have to combine it with proprietary modules later on.

But may I reach out to you privately about possible options? Contact me mark[at]blurbusters.com or at contact me. Either way, congratulations for finding a VRR solution for TestUFO. But I do really need help with the skeleton, and am willing to pay for that bit of help. I've been looking for years for a VRR-compatible web browser wrapper (for TestUFO VRR), and this is the likely breakthrough solution, for custom stutter-free below-max-Hz frame rates with green VALID for any custom framerate=Hz. Perhaps one day of consulting work or something.

• Could be dual-license (e.g. forking me or licensing me a copy via an alternate license; this is legal if you're the only developer)
• Could be rewrite into a simpler skeleton example (just one fullscreen quad, no VRR yet, no UI, relay mouse/keyboard input), which I'll later modify to suit
• Alternate options that might move me forward ASAP

[reitowo]

@nishitha-burman Hi, all. I implemented the feature in CEF, probably you guys got interested: https://chromium-review.googlesource.com/c/chromium/src/+/5265077 https://bitbucket.org/chromiumembedded/cef/pull-requests/734

[aidv]

@nishitha-burman Hi, all. I implemented the feature in CEF, probably you guys got interested: https://chromium-review.googlesource.com/c/chromium/src/+/5265077 https://bitbucket.org/chromiumembedded/cef/pull-requests/719

It says that the pull request has been declined. What does this mean?

[reitowo]

It says that the pull request has been declined. What does this mean?

Sorry, moved to https://bitbucket.org/chromiumembedded/cef/pull-requests/734

[aidv]

Can someone help me understand what is being discussed here?

I'm having a challenge that seems to be quite hard to solve.

I'm looking for the fastest way to push large amounts of data to a WebGL/WebGPU context and render something in the WebGL/WebGPU scene.

The pipeline looks like this: CPU -> IPC -> WebView2 -> GPU

However, the performance is very slow, and there are many points of failure along the way.

Our algorithm runs on the CPU (soon on the GPU with OpenCL) The generated data is sent via IPC to the WebView, which is really slow, The WebView2 runs web code that pushes the data to the GPU as a texture, e.g webGPU....writeTexture()

All of these steps take time.

For example, just pushing the texture data to the GPU takes 1.2s, while the equivalent pure C code takes 1.2ms (!!!!!!).

One idea I had was to create an offscreen C/C++ OpenGL renderer and send the frames to the WebView2, but since IPC is slow, this won't work.

Would what has been discussed in this gihub issue potentially solve the bottlenecks I am having?

What is the absolute best and most efficient way to render something in my C backend and show it in a on my WebView2?

[weltkante]

Would what has been discussed in this gihub issue potentially solve the bottlenecks I am having?

No, this is about rendering the WebView2 itself inside an offscreen surface, so a browser can be displayed e.g. in a 3d environment, got nothing to do with code running inside the WebView2

[reitowo]

For example, just pushing the texture data to the GPU takes 1.2s, while the equivalent pure C code takes 1.2ms (!!!!!!).

You can try implementing this logic with WebGPU Native, and expose some handle to javascript in frameworks like CEF, just keep everything in native. You can create a WebGPU context natively and just expose them in javascript's WebGPU objects.

[aidv]

For example, just pushing the texture data to the GPU takes 1.2s, while the equivalent pure C code takes 1.2ms (!!!!!!).

You can try implementing this logic with WebGPU Native, and expose some handle to javascript in frameworks like CEF, just keep everything in native. You can create a WebGPU context natively and just expose them in javascript's WebGPU objects.

Really?? That would be perfect.

Please tell me how this could be achieved, it’s critical to our application.

I’m willing to pay for help.

[aidv]

You can create a WebGPU context natively and just expose them in javascript's WebGPU objects.

I have no idea how this would be done with WebView2. Any idea?

[softworkz]

I'm looking for the fastest way to push large amounts of data to a WebGL/WebGPU context and render something in the WebGL/WebGPU scene.

The pipeline looks like this: CPU -> IPC -> WebView2 -> GPU

Which kind of IPC are you using? Have you tried ICoreWebView2SharedBuffer? https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2sharedbuffer

[aidv]

I'm looking for the fastest way to push large amounts of data to a WebGL/WebGPU context and render something in the WebGL/WebGPU scene. The pipeline looks like this: CPU -> IPC -> WebView2 -> GPU

Which kind of IPC are you using? Have you tried ICoreWebView2SharedBuffer? https://learn.microsoft.com/en-us/microsoft-edge/webview2/reference/win32/icorewebview2sharedbuffer

Yes I'm using ICoreWebView2SharedBuffer, but it's extremely slow. Edit: It's also synchronous, so it blocks the UI.

Sending 62MB of data from the backend to the webview2 takes about 4 seconds.

It's actually faster to save the data on disk, host a webserver, and use fetch() in the frontend to get the data, about 380ms.

So ICoreWebView2SharedBuffer is very inefficient, but I'd want to access the WebGL/WebGPU scene directly from the native backend if possible, but render the scene in the frontend.

This seems to be quite a big challenge.

[softworkz]

Yes I'm using ICoreWebView2SharedBuffer, but it's extremely slow. Edit: It's also synchronous, so it blocks the UI.

Oh, didn't know it's so bad. Their file access isn't really fast either. Unfortunately they have removed support for pepper plugins (NACL). We still have an application that is using it for rendering videos on surfaces via OGL ES (Electron App using a trick to enable it), but it doesn't work in the Edge WebView. A while ago, I had talked to Chromium devs about this gap (unable to render some, content onto a surface inside the browser), but unfortunately they didn't have a good replacement to suggest. We're using a layered approach now, i.e. transparent webview on top. You could also have another one below...

[softworkz]

It's actually faster to save the data on disk, host a webserver, and use fetch() in the frontend to get the data, about 380ms.

Assuming, your data is image data, how does this compare?

  let image = document.createElement('img');
  image.src = 'file:///c:/my62mbImage.png'
  await img.decode();
  let bitmap = await createImageBitmap(image);

It's non-blocking at least.

[aidv]

It's actually faster to save the data on disk, host a webserver, and use fetch() in the frontend to get the data, about 380ms.

Assuming, your data is image data, how does this compare?

  let image = document.createElement('img');
  image.src = 'file:///c:/my62mbImage.png'
  await img.decode();
  let bitmap = await createImageBitmap(image);

It's non-blocking at least.

This gives pretty much the same performance as fetchibg from a webbrowser. Very fast.

But problem is that this would require some disk io, which isn’t optimal.

You mentioned a transparent webview, I thought about that too, simply create a native opengl scene and overlap a transparent webview on top.

I guess this approach is valid too.

Any caveats to think of if I use this approach?

[softworkz]

This gives pretty much the same performance as fetchibg from a webbrowser. Very fast.

But problem is that this would require some disk io, which isn’t optimal.

Maybe using overlapped (memory-mapped) IO, but when you change the data (which would be just a memcopy), I'm not sure whether the browser engine would read it again, rather than assuming it has cached it already, and I don't know whether it is possible to rename a file which is opened in this mode. Browser caching itself might also involve copying of the data, yet I'm not sure whether this is done for file: URLs.

It's really frustrating when you look through all of those new browser APIs. They have tons of features in many directions, but there's hardly any good and fast way for getting local data into the browser... I've read an article abouit "transferrable objects" in WebAPIs where it was shown that a 32MB blob could be transferred between WebWorker and main thread in 6.6ms instead of 360ms. On Android, it's possible to transfer such transferrable objects between the (Chromium) WebView and Java native code (did that last week for a MessagePort, those don't have much data, so can't say much about performance). Yet, the Edge WebView2 doesn't have something like that. They only have that ridiculously complex mechanism where you need to create COM interfaces for the native object which you want to share with the webview. I'm not sure how that performs, but it might be another option in case you haven't tried.

Any caveats to think of if I use this approach?

WebView Transparency doesn't work in WinUI3 apps and you cannot render WinUI3 content on top of a WebView2 (there's a way though, but with other caveats). Also, CSS filters don't work on the background (because it don't know about). But everything else is fine. We're using this approach on many platforms.

[aidv]

This gives pretty much the same performance as fetchibg from a webbrowser. Very fast. But problem is that this would require some disk io, which isn’t optimal.

Maybe using overlapped (memory-mapped) IO, but when you change the data (which would be just a memcopy), I'm not sure whether the browser engine would read it again, rather than assuming it has cached it already, and I don't know whether it is possible to rename a file which is opened in this mode. Browser caching itself might also involve copying of the data, yet I'm not sure whether this is done for file: URLs.

It's really frustrating when you look through all of those new browser APIs. They have tons of features in many directions, but there's hardly any good and fast way for getting local data into the browser... I've read an article abouit "transferrable objects" in WebAPIs where it was shown that a 32MB blob could be transferred between WebWorker and main thread in 6.6ms instead of 360ms. On Android, it's possible to transfer such transferrable objects between the (Chromium) WebView and Java native code (did that last week for a MessagePort, those don't have much data, so can't say much about performance). Yet, the Edge WebView2 doesn't have something like that. They only have that ridiculously complex mechanism where you need to create COM interfaces for the native object which you want to share with the webview. I'm not sure how that performs, but it might be another option in case you haven't tried.

Any caveats to think of if I use this approach?

WebView Transparency doesn't work in WinUI3 apps and you cannot render WinUI3 content on top of a WebView2 (there's a way though, but with other caveats). Also, CSS filters don't work on the background (because it don't know about). But everything else is fine. We're using this approach on many platforms.

Yeah, it just doesn't seem like there's an easy way to do it. I am however interested in what @cnSchwarzer proposed regarding having a shared WebGPU handle.

This way I could have all the algorithmic stuff happen in the C backend, while the results would be rendered in the WebView2 frontend.

That would be an ultimate solution.

[softworkz]

Yeah, it just doesn't seem like there's an easy way to do it. I am however interested in what @cnSchwarzer proposed regarding having a shared WebGPU handle.

Frankly, I think that's just nonsense.

The statement was:

You can create a WebGPU context natively and just expose them in javascript's WebGPU objects.

WebGPU is a browser API which wraps native platform functionality. How should it be possible to create that "natively"? There is no "native WebGPU context" you can create from outside. And further, there's no WebGPU API which allows you to connect to an outside native context. Neither in one nor in the other direction. An HTML element like would be perfect, which can be part of the DOM but operated from outside...

[aidv]

Yeah, it just doesn't seem like there's an easy way to do it. I am however interested in what @cnSchwarzer proposed regarding having a shared WebGPU handle.

Frankly, I think that's just nonsense.

The statement was:

You can create a WebGPU context natively and just expose them in javascript's WebGPU objects.

WebGPU is a browser API which wraps native platform functionality. How should it be possible to create that "natively"? There is no "native WebGPU context" you can create from outside. And further, there's no WebGPU API which allows you to connect to an outside native context. Neither in one nor in the other direction. An HTML element like would be perfect, which can be part of the DOM but operated from outside...

There’s Dawn, a WebGPU thing for C++.

I’m not sure how it would work, but I’m open to any ideas.

[softworkz]

Well - that IS the WebGPU implementation used in Chromium...

What would be another option is to fork Electron. This gives you access to absolutely everything (i.e. Chromium source level modifications).

[softworkz]

You can create a WebGPU context natively and just expose them in javascript's WebGPU objects.

There's anothher reason why this cannot work: There's a process boundary. There's no way for sharing a (whatever kind of) GPU context between diferent processes. Whatever you do would need to happen within the same process, i.e. the browser's rendering process. Which brings me back to pepper/nacl plugins. These are loaded into the browser process and you can render to OpenGL surfaces directly. Going the Electron way would allow you to keep this enabled until there's a better way.

[reitowo]

I think it is definitely possible to create some modification around Dawn to make it satisfy your needs. There're plenty of Dawn api which is not exposed to js (not in WebGPU standard, obviously), and I was digging around to see how to import a external texture into the dawn context, and find out it is absolutely possible. You'll need to figure out Dawn representatives in v8 engine, then you can do whatever you want once you got them. I cannot give any guarantee, but I think you will not be disappointed if you look into it.

[reitowo]

I was talking about framework (like CEF) level modification from the beginning. This is obviously not a job can be done in WebView2 api. Whatever, it is off topic, and shouldn't continue in this issue.

[aidv]

I think it is definitely possible to create some modification around Dawn to make it satisfy your needs. There're plenty of Dawn api which is not exposed to js (not in WebGPU standard, obviously), and I was digging around to see how to import a external texture into the dawn context, and find out it is absolutely possible. You'll need to figure out Dawn representatives in v8 engine, then you can do whatever you want once you got them. I cannot give any guarantee, but I think you will not be disappointed if you look into it.

Yeah, this veered off topic so I’ma just shut up about it.

But before we put the lid on this, what would you recommend me to look into in terms of dawn and electron?

[softworkz]

I was talking about framework (like CEF) level modification from the beginning

It would require modifications to the Chromium source, not only the framework - which is how I had understood your comment. Sorry for misunderstanding.

Yeah, this veered off topic so I’ma just shut up about it.

Maybe you just create a new issue... (like about content rendering to a panel element in the DOM from the native side)

[mdrejhon]

Missed this, so replying to @fredemmott

Sorry, I wasn’t clear - I have no trouble with VRR 240 in chrome;

To be clear -- I require ability of custom perfect-framepaced arbitrary frame rates below Max Hz.

e.g. 57fps looks exactly like 57Hz, with a green-colored VALID and very flat www.testufo.com/animation-time-graph. Or 123.5fps that looks like perfect stutter free 123.5Hz Etc.

I need a fully dynamic-framerate-capable (not hardcoded to maxHz) VRR framepacing that is correctly refreshing at at dynamic asynchronous refresh cycles, and not to default Windows' scheduled MaxHz-refreshes-per-seconds during DWM+VRR ops. Chromium code design does something to force refreshing of the Chromium framebuffer at every DWM scheduled refresh cycle somehow, spoiling VRRs' raison-detre.

Before W3C moved to WHATWG, I tried to post a suggestion for a VSYNC API that would solve this. https://github.com/w3c/html/issues/375 ... For now, I have given up on browsers' complete inability to do proper true VRR.

So the current fallback plan is to use some offscreen CEF style system, and simply sieze over control over frame presentation to do it the proper way. Hopefully there's no hardcoded tick-tock built into Chromium (Chrome automatically uses 60fps on displays it cannot sync to refresh rate to, e.g. older Linux distributions), or it ruins my plans.

Part of the reason is skills silos -- browser developers are VERY skilled at designing browsers, but don't understand VRR engineering. I believe browser developers don't quite fully understand VRR - so this "browser bug" (non-true VRR during sub-MaxHz frame rates) has existed for almost a decade;

So if I set a 57fps cap (by any means, like busywaits inside requestAnimationFrame() or a provided sanctioned technique...), I need the monitor sending photons to my eyeballs exactly 1/57sec apart (as VRR is designed to do so) and not rounded-off to the next refresh cycle scheduled by Microsoft Windows DWM (even when Windowed GSYNC is enabled). In true VRR operations, the frame Present() or glxxSwapBuffers() call, immediately causes the display to refresh. That's how video games do it.

Why?

1. Display quality can vary at different VRR framerate=refreshrates.
   Engineering Info: Overdrive artifacts (LCD ghosting/coronas) can improve/worsen at different frame rates, due to how overdrive algorithms are optimized for specific refresh rates and not the whole VRR continuum.
2. Over 500 content creators (with ~100 million viewership totalled) use tests invented by Blur Busters. Displays vary greatly in quality and reviewers would love to test different TRUE VRR frame rates in TestUFO without Chrome & Windows force-refreshing MaxHz times per second. For my credentials, see www.blurbusters.com/inventions

Yes, I have tried the framerate cap setting in Chrome, and it doesn't bypass the forced DWM refreshing, unlike many other apps can.

sorry for being unclear: I have no trouble with 240fps with gsync in chrome; that is not translating to the same with WGC+webview2

Desired solution:

• Ability to do arbitrary erratic-stutter-free frame rates below MaxHz with perfect framepacing (as software/hardware allows).

I am still deeply dissapointed that browsers still don't do proper VRR (during sub-MaxHz frame rates).

This is why I am moving my plans to doing offscreen rendering, to workaround this "browsers cant do VRR" bug.

Developer TIP (for Chromium engineers)

Debugging suggestion for Chromium software developers (if anyone reads this):

1. Enable VRR in both your monitor menus AND your graphics drivers
2. Run a browser CANVAS-2D animation in full screen mode (as if you're playing a browser-based game)
3. Turn on your monitor's frame rate / refresh rate setting (most VRR monitors have an OSD that reports current refresh rate)
4. Make sure that the monitor's detected VRR refresh rate matches what shows up in the web browser/diagnostics/your own debug framerate counter.
5. Test arbitrary frame rates inside your display's supported VRR range.
6. TEST PASS CONDITION: For a 240Hz monitor -- if your animation is running at 87.5fps inside the monitor's published VRR range -- your monitor should be reporting 87.5 (true VRR) rather than 240 (automatic compositor in DWM or Chromium)

[Hethsron]

Hi Sorry to come back to the subject, but a lot has happened since it was opened.

And these days, I think there are solutions that should allow you to provide WebView2 on Linux and especially to add this mode.

So, is it possible to provide offscreen-rendering using Ozone Layer ? (cf. https://github.com/chromiumembedded/cef/issues/3263)

The CEF team will be migrating to this architecture. So if, like them, your heart is in Chromium, you can either do what they do or do it better.

Regards

[reitowo]

As you can see in chromium/src/ui/ozone/platform, there's no clue about ozone Windows/macOS support, which I think Chromium/CEF won't migrate to this shortly. In fact, it is also hard to know which platform the OSR host is on, so I think sharing dmabuf is a better solution for now on Linux. FYI, the viz based OSR has been merged to CEF recently: https://bitbucket.org/chromiumembedded/cef/pull-requests/734 I think Webview2 can easily implement GPU OSR using FrameSinkVideoCapturer.

[Hethsron]

Hello @reitolab

I don't think it was merged because this pull request was rejected. The CEF strategy is described here: https://github.com/chromiumembedded/cef/issues/3685 https://github.com/chromiumembedded/cef/issues/3681

What is certain is that Microsoft has what it takes to offer OSR on both Linux and Windows.