Off-screen rendering with webview2

[ajaymonga]

Hi Is it possible to get webview2 to render into a shared memory region like CEF : https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-header-off-screen-rendering

In my application I use 2 process architecture where main process does not have network access and second process uses CEF to render webcontent into a shared memory region from where main app can read the pixels. I am wondering if I can achieve this using webview2

Thanks

AB#28491736

[champnic]

Currently this is not possible. We have offscreen rendering on our backlog, and are tracking it in #20, but I think this ask is clearer so I'll also add this issue to our item. Thanks!

[Harvey3141]

Hi, do you have any updates on this which you're able to share?

[champnic]

Unfortunately not yet. We have not begun work on this yet. This is a large amount of work, and while very high on our priority list, gets bumped each quarter as higher priority asks come in. I really want to do this work as it's currently one of our top asks, but the earliest that could happen is Q1 2022 at this point.

[avail]

Any news on this?

[champnic]

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

[doxxx]

We are currently using CefSharp to render into an offscreen buffer that is sent to an FPGA to be composited onto a live video feed. The render includes playback of MPEG4-encoded video. We were looking into using WebView because CefSharp's default Chromium build does not include the MPEG4 codec. But without offscreen rendering, that's a moot point. We've since produced our own custom build of Chromium to include the codec.

[rjx-ray]

Our use case is to show web content in our immersive VR spaces. See https://www.igloovision.com/software/enterprise-package/igloo-web

We need access to the web rendered textures so that we can warp and blend multiple images to projector outputs to create a clear seamless view on the inside of a cylinder or room.

Currently we use CEF for the web input with a user maintained patch for getting the shared textures. See https://bitbucket.org/chromiumembedded/cef/pull-requests/285 and read down to the end of the comments for full details. We build CEF with proprietary codecs included. The builds of our app and CEF are done with C++.

This is unsatisfactory because it uses the deprecated custom compositing (which could be removed in future) instead of skia and is difficult to keep updated.

We would look at moving to webview2 if access to the rendered textures could be provided and supported along with the ability to include proprietary codecs

[champnic]

Thanks for the info @rjx-ray! If you don't need high-frequency rendering you could consider using the CapturePreview function to get an image, but this isn't great for things like videos or other animations.

[rjx-ray]

Hi @champnic, thanks for the response but we do need high-frequency rendering, typically for YouTube and other web video display.

[avail]

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

Game UI development, specifically within DirectX 9 (Ex) / 10 / 11 contexts.

Ability to render into an offscreen texture and display that as overlay of the game.

[jcain82]

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

Currently, we utilize CefSharp Offscreen for generating a PDF from HTML content or using it to generate screenshots of HTML in a server side environment.

[zziger]

Hi, are there any updates on this feature?

[lencil]

We are starting on the design phase of this work. Would you mind sharing your use case so that we can consider it in our plans?

CAD editor software, I want to compose web ui and opengl/d3d rendering together.

[alainza]

Any XR Application wanting to be able to have some 2D UI inside the 3D Environment: it means being able to render into a DirectX Texture and to be able to inject input in the off-screen view (pointer + keyboard)

[champnic]

Thanks for the info! Unfortunately our design work is slow going due to high priority bugs, but we're still making progress.

[danielvandenberg95]

Is #579 related?

[honzapatCZ]

My use case would be Game Overlay. And actually dotnet/maui(and in extension WPF/WinForms)'s BlazorWebView could benefit from this change too as they wouldn't have to create the fake elements(they are just precisely tracking the WebView window on top), but could do normal real elements in their respective apis without any weird quirks and bugs.

[honzapatCZ]

I'd say an api like CefSharp would be quite pleasant and easy to port over existing CefSharp code. https://github.com/cefsharp/CefSharp/blob/d8674fd076c021eddcc0cb579687ca3c51a63767/CefSharp.OffScreen/DefaultRenderHandler.cs

[Harvey3141]

Hi, do you have any updates you're able to share?

[VentuzTammoHinrichs]

Adding another use case: Presentations on screens that are not just Windows desktops.

From our experience with CEF we'd need the following (in descending priority order) to switch:

• Rendering into either a raw CPU buffer or into a fully accelerated DXGI surface (higher level APIs optional but this is what most people use)
• Full control over the presentation parameters, such as resolution, frame rate, display scaling, output color space (including WCG/HDR), etc. Fully manual VSync is appreciated; the client engine might render ahead and buffer, or batch render to disk, so it's not strictly in real time.
• Proper graphics resource lifetime management. Client app must be able to exactly specify when/how it is able to receive frames and when it is finished processing them; no spurious callbacks when client isn't ready (anymore), and no mutexes that render either side unresponsive when held a bit too long or not acknowledged (that one probably came out a bit too detailed, but you might be able to feel the pain here).
• Full manual input injection, such as keyboard, mouse, and multi touch inputs.
• Audio should also be redirected. Ideally per browser instance but globally would be fine at first. Client must be able to specify the sample format and channel layout, and it should be a pull model in which the client requests a certain # of sample frames and WebView2 outputs that number of frames exactly, in order to avoid timing discontinuities between the browser audio and the actual audio path the client uses.
• ... and you might recognize a pattern here. There are other subsystems that would be cool to have redirected to an API, for example audio/video input (for web based streaming/conference services) or geolocation. A feature to auto-block all outside interaction that's not intercepted (Midi, Bluetooth, notifications, printing (possible security hole there!), etc) would perhaps be helpful, too.

It might be tempting to keep these APIs as close to the corresponding Windows APIs as possible but it's not strictly necessary - this feature will mostly be used from deep within client code, and very possibly behind a bunch of abstraction layers, so I'd aim for minimal and clean first.

Hope this was not too much/harsh but currently a whole industry is dependent on that CEF pull request above which has been in PR limbo for years and will soon just stop working altogether, and an alternative to that would be a very appreciated thing ;)

[DanielsCode]

@champnic - Is there any news you can share with us?

[robotsinthesun]

+1

[Curve]

+1

Please use the thumbs up reaction instead of occluding the issue with +1 comments

[champnic]

@DanielsCode Unfortunately not really. We had begun the design phase but are currently dealing with high priority bugs. I really want to get this done as I know it's a huge pain for a large portion of our developers, and I'm hoping we can get back to focusing on it soon.

[LukeTOBrien]

Hello, just to add another use case.
I would like to create a ASP Web API controller where the user POSTs an HTML string, or JSON object that is merged into an HTML template, the controller would use WebView2 to render the HTML off-screen, save a PDF to a stream and then return the stream as a file for the user to download.

It seems CoreWebView2.PrintToPdfStreamAsync is in pre-release, so we just need headless mode and I'm good to go.

[Karbust]

Hello,

Using WinForms, has anyone tried to put the WebView2 component on a separate Form and put that form off-screen or make it not Visible?

With either of this code:

        private void OnFormLoad(object sender, EventArgs e)
        {
            Form form = (Form)sender;
            form.ShowInTaskbar = false;
            form.Visible = false;
        }

        private void OnFormLoad(object sender, EventArgs e)
        {
            Form form = (Form)sender;
            form.ShowInTaskbar = false;
            form.Location = new Point(-10000, -10000);
        } 

Was able to use it this way but, since it's not an "official" way, anyone that has used it has detected any kind of issues with it?

The first bug I noticed is when doing Alt+TAB with the form on offscreen it shows up on the list, with the window not visible it flickers when it opens.

Thank you

[pzxbc]

+1 wait for off-screen rendering

[bbday]

yes should be great off-screen version like cefsharp to automate task with clicks/keyboard input and intercept request/response

[nishitha-burman]

Hi all,

We are investigating this request and have a couple of questions to better understand your scenario:

1. What framework is your app (e.g. Win32, WPF, WinForms)?
2. How does your app render it's individual elements?
3. Would rendering WV2 to a ID3D11Texture2D or IDXGISurface address your scenario?

Thanks, Nishitha

[honzapatCZ]

Game Overlay 1) .NET 7 2) DirectX/Vulkan, getting bitmap buffer would work cross-api 3) 50%

[avail]

DX9/DX12 Currently CEF yes.

[nishitha-burman]

@honzapatCZ and @avail just updated question #3. Thanks!

[avail]

To update answer 3; being able to handle OnPaint for DX9 to at least have software rendering would be nice, but a texture/surface would definitely be extremely useful for the more modern rendering backends

[justinstenning]

@nishitha-burman 1) .Net 6/7 and Win32/C++ 2) currently CEF and DirectX/Vulcan/OpenGL 3) Yes

[rjx-ray]

1. Win32, OpenFrameworks
2. CEF, shared textures patch, d3d11 texture->interop ->OpenGL
3. Yes

[LukeTOBrien]

1. ASP.NET Web API
   Save to PDF service using CoreWebView2.PrintToPdtStremAsync
2. Blazor WASM app, does not render WebView2, PDF stream returned from controller in File method
3. N/A

[bbday]

1. .NET 7 WPF
2. DirectX/Vulkan, getting bitmap buffer would work cross-api

[nashilnik]

1. .NET 6, WPF with a custom Direct2D canvas
2. Direct2D render target. I'd expect the following or similar flow: WV2 -> IDXGISurface -> ID2D1Bitmap -> ID2D1RenderTarget
3. Yes, as long as IDXGISurface is shareable

[doxxx]

1. .NET 4.7.2 service (no desktop GUI), may move to .NET 6/7/... in the future.
2. We're only using the CefSharp browser to render.
3. I think it would be difficult. We're currently using the CefSharp Paint event to take the raw 32bpp image bytes and send them to an FPGA for compositing onto a live video feed displayed on a dedicated monitor.

[kebby]

1. C++ (win32, Direct3D11) + C# (winforms)
2. CEF with shared textures patch and ChromiumFX -> D3D11
3. Yes.

[salvadordf]

I'm the maintainer of WebView4Delphi and CEF4Delphi. I can't speak for all the developers using WebView4Delphi but I've been asked if it would be possible to add an off-screen rendering mode to WebView4Delphi.

1. Win32 using Delphi or Lazarus with WebView4Delphi.
2. Copy the bitmap buffer for maximum compatibility.
3. That wouldn't be enough. Please, consider using a bitmap buffer in addition to a ID3D11Texture2D or IDXGISurface.

[martonp96]

1. C++ (win32, Direct3D11) + C# (winforms)
2. CEF with shared textures patch and ChromiumFX -> D3D11
3. Yes.

[thomasreiser]

1. C++ (Win32, Xbox)
2. CEF -> update texture (BGRA8) in CEF's OnPaint event
3. Yes

[Devyre]

1. C++. Desktop cross platform. d3d11, opengl/vulkan.
2. CEF with shared textures patch + custom modifications.
3. As long as the texture can be shared it would be good for win32.

[2A5F]

1.Unity 2.DirectX 3.Yes

[yewnyx]

1. C# .NET 7 Console App on Linux or Windows
2. Software rendering (i.e. save to png on demand, resolution ideally settable on request)
3. N/A or probably not

[Robula]

1. ASP.NET Web API (C# .NET 7)
2. Currently, using PuppeteerSharp to render SVG and convert to PNG/JPG. Interested in WV2 as a replacement.
3. N/A

[dt200r]

Any update on this? Looking to offscreen in .NET 7 asp.net.

[paulhobbel]

We use it as game overlay

1. Win32
2. DX11/12
3. Yes that would be perfect

[mdrejhon]

1. Flexible, but using C#/.NET with interop calls where needed.
2. Flexible, but I am familiar with MonoGame engine.
3. That would be an acceptable workaround.

I am researching integrating my popular display motion testing website, www.testufo.com -- into a Direct3D executable, that can run offline, and is capable of custom frame rates to simulate custom refresh rates (on VRR displays such as G-SYNC and FreeSync).

Note: I get millions of unique visitors; it's the most popular display motion testing website, and many content creators use it as part of their gaming monitor benchmarking test suite. There are over 30 different tests selectable at upper right. Combined with configurable options in each test, this creates over 1 million possible variations of different TestUFO tests or educational demos (some animations are for testing, and some animations are educational about display science).

I need to control the compositing rate (refresh rate / paint rate) too.

AKA, simple terms; I'd like to have one browser compositing event per Direct3D frame rendered. For all animations, paint updates, scroll updates, graphics updates, etc. And have the requestAnimationFrame() javascript callback event synchronize correctly to the frame rate.

When you add WebView2 support to Direct3D, make sure it is capable of being configured to synchronize one browser compositing event per one Direct3D frame presented (for browser animations/scroll/paint events) -- even independently of the display refresh rate.

Why?

(See More) Details and Use Cases

## Short Version: I have additional requirements; My target display isn't necessarily the same refresh rate as the physical displays. In addition I also need to simulate a custom fixed Hz on a VRR display (e.g. simulate 100Hz via outputting 100fps VRR to a 144Hz display). So that things like requestAnimationFrame() within the WebView2 will automatically synchronize to the frame rate of the Direct3D -- requestAnimationFrame() is specc'd to occur once every refresh rate as long as performance & spare processing headroom & power plan permits. ## Long Version: I need to see www.testufo.com/refreshrate and www.vsynctester.com successfully detect that a custom accurately-framepaced frame rate is a specific refresh rate -- the algorithms achieve it by detecting the consistency of time between individual JavaScript requestAnimationFrame() (the defacto HTML5 refresh cycle callback event) during a low-processing-overhead animation. It should be able to detect it as a refresh rate if it's accurately framepaced (e.g. 73fps framepaced almost exactly 1/73sec apart, best-effort). Basically the browesr compositor loop needs to be able to sync to the frame rate of the DirectX engine, so that requestAnimationFrame() within the Javascript inside WebView2 synchronizes to the frame rate. When the frame rate is framepaced accurately (+/- 5% jitter) TestUFO recognizes it as an assumed 'refresh rate'. TestUFO has multiple millions of unique visitors quarterly, and is used by over 500 content creators representing a net total of over a hundred million of viewers/subscribers (e.g. RTINGS, LinusTechTips, TomsHardware, C|Net, etc) ([list of some content creators using my TestUFO](https://blurbusters.com/motion-tests/reviewers-using-pursuit-camera)) as part of their display/gaming monitor testing suite. So even more people see the results than even visit the site directly! Since display quality is different in VRR versus non-VRR mode, I want to use WebView2 + Direct3D + custom frame rates, to run TestUFO in VRR mode (via custom-set frame rates). TestUFO is designed to perform well at even high refresh rates, as long as GPU acceleration is available. It even worked fine at 480Hz on an experimental 480Hz monitor, [photos here](https://blurbusters.com/480hz) a few years ago Essentially, the virtualized 'refresh rate' of WebView2 can just be the frame rate of Direct3D that's displaying the WebView2 as a texture. Custom fixed frame rates during VRR operation (which is possible by Direct3D), has been a very frequent feature request by content creators responsible for millions. There may be a situation where you need to let the Direct3D developer configure whether they want WebView2 compositing events at: - At frame rate of the Direct3D (the frequency of the Present() event); or - At custom fixed rate (60 by default), independent of Direct3D; or - At the actual display refresh rate (of the monitor that the Direct3D context is running on). - Or enable an optional callback event, that executes the next browser compositing event and its related callbacks (requestAnimationFrame etc) These flags/arguments will help hint how frequently the WebView2 should composite -- for things like animations, scrolling, and other things move. I want to composite WebView2 at a rate matching the Direct3D, but that can interfere with animations within it (especially during varying frame rates). There are pros/cons for each; requiring configurability. ## Example Use Cases - A VR or AR headset may render at a refresh rate (90Hz) that is different from the physical displays that shows up in Control Panel (60 Hz). The developer would configure a custom fixed rate, or use the callback event. - A developer may want to keep animations in webpages running at a constant rate independent of the Direct3D rendering rate. In this case, you'd choose a custom fixed rate, or choose actual display refresh rate, as the WebView2 internal compositing frequency. The browser offscreen framebuffer would keep refreshing continuously at its own independent frame rate, to be blitted to a texture once per Direct3D frame. - Or other developers may want to go more 'efficient' (power saving). Frame rate would be the method of power management for such app. Low frame rates such as 30fps is more miserly than high frame rates. - Offscreen rendering may not wish to be stuck at 60, and such a developer may have a custom 'refresh rate' they want to simulate -- whether low for processing-efficiency or high for making scrolling and animations work optimally matching a performance display (e.g. 120Hz phone, tablet, xbox, television etc, in the semi-mainstreaming of 120Hz). Whether as a texture or a HUD overlay or etc. - Or more 'custom' (intentional custom frame rates) , by having only one browser compositor event occur per Direct3D frame. (and its attendant JavaScript-level callback events such as requestAnimationFrame() ), for animations that are intentionally designed to treat a frame rate like a refresh rate. (This is actually technically true for VRR -- the frame rate is the refresh rate, and the refresh rate is the frame rate -- whenever the frame rate is inside VRR range, the display immediately refreshes upon frame presented by the computer. Instead of having its own refresh rate clock, VRR such as FreeSync and G-SYNC means display is syncing to the frame rate -- ala www.testufo.com/vrr simulation) ## Example Precedents in Experiments The best browsers running on AR/VR headsets sync their compositing refresh rate to the refresh rate of the headset, whether it's standalone, or whether it's casted (e.g. from a PC). It stutters very awfully (and stutters = headaches) if the browser compositing rate isn't synchronized. I was able to trick a Chrome window or Edge window to sync to a VRR rate in a very approximate (rough) manner -- e.g. windowed VRR mode with a Chrome/Edge window running in the background. You need to use NVIDIA Control Panel setting for windowed VRR where VRR is enabled for the foreground windowed app. When this is done, and you play with this, this forces all background apps to composite at the same rate as the foreground app. The foreground video game window would fluctuate in frame rate, and force the compositing of the underlying browser window to vary. (With some minor side effects). In my situation, I would intentionally run a constant Direct3D frame rate, in order to create custom "compositing rates" within Webview2 -- and I would want to configure compositing to match the frame rate of the Direct3D presentation. Meaning I need browser compositing events (and its underlying callbacks) would synchronize to the frame rate, causing its requestAnimationFrame() callback event inside Javascript to occur once per Direct3D frame. I was evaluating the use of CEF, and synchronizing its paint event to do the workaround, but WebView2 would be quite fantastic.