HUD over render in webgl

[Huck]

Hello,

i am interested in creating a HUD in webgl. I know html can be used to overlay over the scene, but having the UI accelerated on the card can have some mighty advantages, mostly integration with the rest of the app, 3D transforms/transparency and shader special effects.

I was wondering if there was a way to dual render a scene and blend/overlay it. For example to render a a first time with a perspective camera, and then render a second HUD layer on top with an orthographic camera and blend it with the first.

thanks

[Huck]

related to this i also have a question. Knowing that html compositing over the webgl canvas is done on CPU (ouch!), is it more efficient performance wise to double render (ortho over perspective)(render-to-texture perhaps?) or to let the browser do it on CPU?

thanks

[alteredq]

Knowing that html compositing over the webgl canvas is done on CPU (ouch!),

Not really. At least on Chrome and Firefox page compositing is done on the GPU, already for quite a while.

[Huck]

that is REALLY good to know! thanks :) so i guess if this is, the cost of html/webgl compositing is good in a modern browser?

[alteredq]

Yup. There was a marked increase in performance when they changed it.

[Huck]

i managed to create the effect i wanted by creating 2 scenes and 2 cameras, one perspective and one ortho. I then render the first scene with the first camera, setting autoclear on the renderer first to clear the surface. Then, i turn off autoclear and render the second hud scene with the second ortho camera, effectively compositing both, the hud over the other.

now to the experts out there, would this technique of double render be more efficient (performance wise) than using html and letting the browser composite the whole thing in GPU or is there really no gain at all and using html/css over the canvas is just as good? (again the goal is to maximise fps performance)

many thanks :)

[Huck]

actually, i was just thinking of another potential solution. i guess we could overlap a canvas2d over the canvas3d and then draw on the canvas to do the "ortho" hud.

now the question becomes, how does performance compare between:

• 2x render in pure webgl (ortho over perspective)
• html/css compositing over the webgl canvas and let browser composite
• draw on canvas2d over webgl canvas3d and let browser composite

interesting to know for design choices,

thanks!

[mrdoob]

I think you'll need to test and try. But I suspect the 2x render in pure webgl would be the best option. The last time I asked a question like that to a browser developer the answer was that I should try to rely in the browser as least as possible.

[Huck]

it makes sense, thanks. i did some non scientific testing of html divs over the 3d canvas and it does seem to hit fps performance a lot, so i guess pure webgl is the way to go.

[alteredq]

See @toji's answer here:

http://stackoverflow.com/questions/10703212/recommended-way-to-make-a-2d-hud-in-webgl

I'm surprised you got big fps hit. That's wasn't my experience so far.

How fast are these for you (which browser / OS / GPU)?

http://alteredqualia.com/demos/1k/balls/ http://alteredqualia.com/argon/ http://alteredqualia.com/bugs/

While neither of these use WebGL canvas, they use compositing a lot (including on 2d canvases) and at least on my system are very fast (Chrome / Windows / Nvidia and ATI).

[Huck]

well like i said, these are non scientific tests, just objective, but for example this site (http://chrysaora.com/) as soon as i enable the panel i lose about 10 fps right away, as soon as i collapse it, gain another 10 fps. so clearly the compositing is causing hurt. Another one is (http://idflood.github.com/ThreeNodes.js/public/index.html#example/spreads1.json). as soon as i drag the nodes below the render window (causing compositing) the fps becomes very choppy (if the node doesnt touch the render window (hence no compositing, all is fine)). stop dragging and fps is fine too.

now these are only true for me in firefox 12. chromium handles them fine, so who knows. (ubuntu 64 bit, nvidia quadro 2000 + binary drivers).

btw im the one who asked this question on stackoverflow ;) looked on google and there is suprisingly very little info about compositing performance, so thought id ask. took quite a while to get an answer so it seems few people really know the cost of browser compositing.

again disclaimer, this was all non scientific, perhaps real benchmarks could help, i dont have time

[Huck]

p.s.: your 3 examples you sent me ran fine for me in firefox too

[alteredq]

Hmmm, this is interesting. I have exactly the same GPU like you, Nvidia Quadro 2000, just I use Windows 7 64 bit.

I see absolutely no effects on performance in Chrome, though I do get ~5 fps drop in Firefox.

So I guess Firefox compositing is still not that good :/

[Huck]

yeah, for the record i figured i would start with html/svg for the hud as it is the easiest one to do and watch how it goes. if i get performance issues, i will rebuild the hud into webgl later (would take more time due to manual management of text/fonts and container management) but darn, can do some pretty cool 3D effects too ;)

[toji]

Sorry to hijack a closed issue, but I was curious about the performance characteristics of this. I threw together a quick HTML HUD over WebGL demo to give it a try: http://media.tojicode.com/webgl-samples/hud-test.html

Admittedly that's a REALLY simplistic case, but I don't see the overlaid elements having any noticeable performance hit at all on Chrome or Firefox. I'm willing to take suggestion for how to make it more real world, though.

[alteredq]

@toji Measuring is exactly what's needed here ;)

Unfortunately it seems your test throws an exception on Firefox here (FF 12.0, Win7), so it just freezes:

[03:21:31.334] performance.now is not a function @ http://media.tojicode.com/webgl-samples/js/util/gl-context-
helper.js:95

And as for "making it more real world": do you also observe slowdown in Chrysaora on your machine?

http://chrysaora.com/

So far we have covered Linux and Windows 7, where on both there is a slowdown in Firefox but not in Chrome (when you turn on "debug console").

Situation there: fullscreen or almost fullscreen WebGL canvas and then over this overlaid relatively large DOM container element (bigger than screen size so there is a scrollbar) with quite a lot of DOM children.

Also it seem this container element has transparent background while children are opaque.

[mrdoob]

I can confirm the Linux results by the way. On my ancient system the chrysaora example runs at 15fps in Chrome with and without UI. And in Firefox runs at 8fps without UI and 4fps with UI.

[Huck]

Hi Toji, like alteredq mentioned there is a bug in your demo in firefox. im very interested to see how firefox behaves in this case, think you could fix it for posterity?

i realize that although chrome is fine, firefox seems to have serious compositing difficulties. since i need quite a bit of stuff on top of the 3D as weel as animations, i guess 3D will be the only way to go the safe way. html is nice, but its no use if it kills framerate for most. wish i could be proven wrong though, because font handling and managing windows and child controls in opengl is a pain,. as toji mentioned.

[toji]

Wow... that's a really bizarre bug! On my system it runs for about 5 seconds and then poof my shim for performance.now simply disappears. Weird! I've replaced those calls with Date.now(), which fixes it, but I'm certainly going to be looking into that more.

[alteredq]

Wow... that's a really bizarre bug! On my system it runs for about 5 seconds and then poof my shim for performance.now simply disappears.

Just a guess - performance built-in object seems to be tracking also page loading / DOM construction / etc.

Maybe your shim is added to one instance of this object, but then when some page building events finish, Firefox just creates a new instance of performance object (instead of updating values of existing object) and your shim doesn't exist there anymore.

Also now I can confirm no observable slowdowns with your test on my system in Firefox.

Maybe you could test:

• make GL area full window
• add more children DOM elements to HUD
• make HUD element with overflow (to get scrollbars chrome)
• add event handlers to DOM elements in HUD

[toji]

Your theory about the performance object seems pretty logical, though if that's the case then the behavior has changed in the FF nightlies. I'm not sure if there's anything that can be done to fix it for the current stable build if that's the case, though. :(

FYI: The little Log window in the upper left already overflows, so that's one case down. The rest of your suggestions are all good ones, and I'll see if I can work some of them in.

[alteredq]

I meant HUD element overflowing the whole window height, like in Chrysaora.

It may be internal DOM renderer that it's causing slow down (we usually don't measure framerate for just HTML pages so I have no idea how expensive this can be, though there are certainly some pages that feel slower than others).

[Huck]

Toji, i see that you fixed the demo for Firefox, and results are interesting to say the least.

WITHOUT the hud, i average 62fps. when i turn ON the HUD (with ALL effects activated), i go all the way up to a range of 80-89fps... if i turn OFF transparency, i go down to a range of 70-80fps. turn if back on and back to 80-89. turn HUD off and back to 61.

chrome stays perfectly constant in all cases at 60fps.

weird for firefox...

[toji]

What version of Firefox and Chrome were you testing with?

[Huck]

FF 12, ubuntu 12.04 64.

and yup, i confirm this happens to me consistently. just tried at least 15 times in disbelief, happens consistently. firefox is messed up in its compositing

[Huck]

perhaps this is related.

https://bugzilla.mozilla.org/show_bug.cgi?id=724476

i just tried firefox 10 and i do have a slightly better performance overall. so its true for me that FF12 webgl is slower than FF10. and again even in FF10, i get much better performance with the HUD ON, like 20 fps more. weird i know...

[Huck]

and just to add on this, i just tried FF13 beta4, and i get much improved webgl performances. on Toji's demo, i get a stable 60fps without HUD, and with hud i go up to 68fps no matter which options i choose as long as the hud is visible, higher fps. but overall it feels more stable, much less jumpy.

i also get drastic improvements in Chrysaora demo (about 20fps gain over FF12), although there is still a 10fps drop for enabling the HUD. so its a partial gain...

[mrdoob]

1979

[djone010]

We are seeing this firefox performance issue very bad on our project which has dojo UI elements floating over the webgl canvas. Chrome handles this like a champ and firefox drops ~10 FPS for every overlayed UI element. What's interesting is that opening firebug also dramatically decreases the frame rate too. The frame rate stays crippled even when the firebug console is closed again. You can try this on the http://chrysaora.com/ example. It does not get better again unless I totally deactivate firebug. This is not an issue for chrome and the chrome debugger. I mention this because the debugger is just a fancy web user interface and this shows that firefox cannot handle rendering both a webgl canvas and a fancy dom based UI at the same time. This is the first place that I found of anyone else noticing the issue, so I thought I would chip in with what I am seeing.

[anothercoder2]

Not sure if anyone has looked at this http://www.zephyrosanemos.com/windstorm/current/live-demo.html There is some explanation of how it was built. http://www.zephyrosanemos.com/ Thie GUI is built using WebGL and is way superior then anything I have seen out there. Would love to something like this added to three.js

[vthibault]

Since I tested both for my project, I can give some feedback as the issue is 2 years old, I just focus on cons :

HTML

• It really sucks to animate a cursor. In my case I had to build all animations of the cursor in a canvas (one canvas per animation), get back the data URI of the canvas, decode the base64 to get the image content, store it in a blob url and use the blob url in the rendering loop to bind the css cursor with the url.
• Need to check that your html/css is cross browser (few work).
• You inherit browser problem. For example it's a big pain to work with a scrollbar in a draggable UI element.

WebGL

• Have to rebuild a whole HUD system with event handler.
• If your app run at 5 FPS, the UI will not be responsive and that sucks ! That's not the case of the HTML still smooth even if you have 1FPS.

Canvas

• Same thing than WebGL.
• Not so fast.
• The font rendering is really ugly and is browser AND GPU dependent.

---

I personally decide to use HTML, and to be honest I have some complexes elements and I don't see a boost penalty. In all case it depend of what the dev need to do.

[anothercoder2]

hi vthibault, I guess my request is a library for building a HUD easily with powerful features. In this link http://www.zephyrosanemos.com/windstorm/current/live-demo.html The hood as features such as 1) Drag-gable panels. 2) Tabs (with effects) with the HUD 3) Minimize/Maximuze 4) Cool progress bars 5) Menu system 6) Ability to embed webgl content with the panel e.g the External View HUD in the example above. 7) Theme that blends with the scene and probably ability to use shaders etc I have not seen anything with three.js that comes even close. This probably needs to be a extension library. Any serious application would probably need a HUD. From a three.js user perspective it would be great to have a HUD library/api available instead of focusing on building it from scratch. If this request is on the wrong forum please let me know where to post this and I would be glad to do so. Also if there is a example that is even close to the features listed about using HTML5/CSS it would be great to see it.

[kumavis]

@anothercoder2 wow thats a great demo

[jaamo]

I was working on a project using WebVR / Oculus Rift and the DOM solution wouldn't work. I created the HUD with pure Three.js.

I created a separate scene and rendered that on top of my original scene. I used 2D canvas to dynamically generate a HUD texture.

You can see the live example here: http://codepen.io/jaamo/pen/MaOGZV

And blog post about the solution here: http://www.evermade.fi/pure-three-js-hud/

[mrdoob]

@jaamo 👍👍👍

[WestLangley]

@jaamo For reference, the OrthographicCamera constructor parameters should be in pixels only if your world units are in pixels.

Notice that your HUD text fills the screen if the window size is reduced.

See this post, and a HUD example in http://threejs.org/examples/webgl_shadowmap.html.

[ghost]

@jaamo I just tried using using your solution with the latest three.js (r73) and it stops updating the canvas. Any thoughts as to why this happens? It's as if it ignores "hudTexture.needsUpdate = true"

I've updated the three.js source in this codepen: http://codepen.io/waichuen/pen/ojMJKB

[manthrax]

Just a note to any interested parties.. you can set the css element z-index to one behind the gl canvas, and punch through the alpha of the canvas using a duplicate proxy mesh for the css object.. then you get actual 3d ui, in your world with proper zbuffering, etc. I've gotten this working in a virtual arcade enviro, and it looks awesome.. unfortunately, at oblique angles, the css objects flicker in and out, or disappear.. if I disable scene.render on the gl, they reappear. Has anyone seen this behavior?

[manthrax]

This is r76 btw.