mdrejhon
commented
4 years ago Updates, edited the above to add more information:
-
If you're using Linux (GPU performance varies much more than with Windows) -- make sure to test using GPU-accelerated web browsers; sometimes this requires driver configuration & chrome://gpu verification. The 50-to-60 de-stuttering standards conversion doesn't work well in non-GPU-accelerated browsers that aren't able to VSYNC. If you have no Windows/Mac machines, try your iPhone or Android; although an iPad or Galaxy Tab works better for those large TestUFO animations -- those will run the dmonstration standard-convert algorithm just fine.
-
Add mention to theoretical GPU shader processing which now becomes an option (an offscreen canvas bitmap can be passed to a WebGL as a texture for shader processing)
-
New filter idea added: With the added filter hook, it now becomes feasible to have a working Emularity canvas to be mapped to a WebGL arcade machine, or a virtual reality arcade in WebXR -- with actual playable arcade machines you can walk up to while wearing a VR headset. Now that's true Emularity to the namesake!
NOTE: @db48x -- If you have tried out a new Oculus Quest 2 standalone VR as I have, you'll quickly realize this is already practical -- Quest 2 is 1,000x better Holodeck immersion than Toy VR ViewMaster or Google Cardboard -- albiet encumbered by the Facebook requirement -- but it shows the potential of walking into an actual virtual reality arcade, written in standards-compliant web programming! And its in-VR browser already supports WebXR and is already powerful enough to run the Emularity emulators at Internet Archive already (via its built-in overclocked fan-cooled Qualcomm Snapdragon 865 GPU), although there's no keyboard to control them unless you pair an external Bluetooth keyboard. Also, with SideQuest sideload, I can force a 60Hz refresh rate (low persistence strobed -- CRT clarity mode), which means Sonic Hedgehog scrolls perfectly CRT-clear zero blur on the virtual floating screen in VR. Ideally, I'd like Emularity to someday add support for Oculus thumbstick joystick A/B/X/Y buttons via the WebXR APIs, though this could theoretically be outsourced to a different javascript wrapper that calls specific Emularity APIs for controller input -- is there already a speedrunner/roborunner Emularity API of any kind for accepting API calls to simulate controller input? If so, then one can write a 50-line-to-100 line JavaScript wrapper (utilizing some existing toolkit such as webvr.js) that doesn't need any modifications to Emularity main base itself. Then I'd be able to test-out playing those Internet Archive games at least on a floating browser window inside my Oculus Quest 2 headset, using just the Oculus GPU, without a computer, and controlling the already-able-to-run Emularity emulators using my Oculus controllers! Niche case, but it's so weird everything runs perfectly and at full speed (most of the emulators) in a floating browser window inside my VR headset, but unable to play them using the joysticks I'm holding in my hands!
Nontheless, Emularity is a pretty impressive piece of work and I realize large parts of it is not really actively being maintained at the moment because of difficulty porting non-JS emulators to JS -- but a theoretical uptick of maintainers could arrive, with the simple add of filter hookability and controller hookability -- a quick glance suggests this is a ~300-to-500 line change to Emularity (including one simple example filter for demonstration purposes).
For now just KISS and make Emularity capable of offscreen canvas, so it can be part of a filter chain to an onscreen canvas. It will unlock infinite possibilities in the era of improving browser capabilities/performance.
My quick look at the Emularity source code, particularly loader.js, is that no changes to the emulator modules are needed and the emulator framebuffer object is directly compatible with an offscreen canvas / context to Bitmap object (it's actually the same object type) -- the very same object capable of highspeed-JS-processing capable with the bitmap mathematics APIs.
There are three main workflows I can see potentially compatible with the Emularity workflow:
-
Context to a hidden CANVAS element, perhaps a CSS hidden style. Emularity copies it to Bitmap every emulator refresh cycle, for reprocessing to a second CANVAS element that is visible.
-
Context to a Bitmap object. Emularity saves the bitmap object every refresh cycle, and a new bitmap object used in its place for the next emulator refresh cycle to render to. This requires a little more rearchitecturing, but improves efficiency a bit.
-
Context to an OffscreenCanvas object. Emularity will easily work with this because it's a drop-in replacement in loader.js so rearchitecture requirements become nearly zero;
- 1 and 3 are easiest for Emularity requiring only tiny modifications
- 2 provides more efficient performance (passing contexts instead -- Canvas object contexts and Bitmap object contexts) since this is the common data format between Canvas and Bitmaps
- Both 1 and 2 and 3 can use a rolling queue of preallocated Bitmap objects can be used to prevent garbage-collection overheads (of repeatedly allocating/deallocating framebuffers).
- Both 1 and 2 and 3 are the same object (same instanceof!), as canvas.getContext() directly maps to a bitmap context, so all the same JS APIs can filter reprocess.
For memory-constrained browsers, one can point to the onscreen context, bypassing the offscreen (the "filters disabled" mode). Either route works with any of the above.
Some niggly small amounts of refactoring appears to be needed, but nothing too killer seems to be needed.
Metaphorically, it makes logical sense, because we're simply fundamentally software-emulating a display, and a display input is fundamentally a unidirectional receive-only device; so it surmises that no upstream communications are naturally needed. Framing programmerthink this way; makes it much easier to conceptualize filter processing as a black box that the emulators don't need to know about (much like the processing in a display's motherboard).
Supporting filters is simply Emularity embracing its purpose: The ability to use a browser as the display output. We're just "enhancing" the emulator's display.
So I feel reasonably confident it is not a megaproject to at least "enable filter hooks" (even if you don't write the filters). Maybe no programmer resources can do it this year, but I think this should be done by 2022-2025 -- even as early as 2021.
Historically, Emularity has outsourced filter processing to the individual emulators.
However, Emularity is a metaphorical equivalent of an emulated monitor. It allows a web browser to "become the emulated display for emulators".
Thus, Emularity theoretically takes upon responsibility of a monitor's motherboard and its processing algorithms. This github suggestion treats Emularity accordingly in this perspective.
A real display/monitor/television is able to process refresh cycles for things like color processing or scaling. A real display/monitor/television does things without the original game or video source knowing.
Thus, for an emulated display, this oneway capability is easily blackboxable and siloable. Thus, Emularity (emulating a monitor) can add filter-processing layer without needing any underlying modifications to individual emulators. It has lots of long-term pros:
Add Emulator Framebuffer Processing Filter Hooks To Emularity
This is a natural improvement for Emularity, given Emularity allows a web browser to be the emulated display for an emulator. Thusly, Emularity acts as a display conduit, with display responsibilities. This github improves and future-proofs the display responsibilties of Emularity in the light of increasingly-GPU-capable web visitors.
The Good News: It's a bit easier than thought: Hidden "CANVAS"
As the author of TestUFO, I have extensive Canvas2D knowledge. And Emularity also uses the HTML5 <CANVA> element. Which is brilliant news: One can render to a memory bitmap instead of the screen. Or point it to a Bitmap object: You just point to a different context -- the emulator renderer doesn't even need to know it's no longer drawing to a visible browser element, but essentially an offscreen element that's mapped directly to a high speed bitmap that can be processed really fast in JavaScript. So this is probably a simpler change than one thinks.
Metaphorically, it is kinda like an offscreen CANVAS tag. Your hacked MAME merrily runs to it, and Emulatity just reprocesses these as high-speed bitmaps to be displayed to the actual visible CANVAs. The algorithms I am now able to come up with, now all fit in the GPU horsepower of even a 5-year-old Intel GPU -- even a mere smartwatch GPU now has enough power for some of the algorithms, amazingly enough;
This makes it easier to add a middleman renderer to Emularity since it only requires unidirectional data flow -- no need for upstream communications (telling emulator to mute or whatever).
It might not be high priority now but this should at least be masterplanned this decade, as people buy 4K displays -- higher resolution displays -- or higher Hz displays -- which can benefit from moving some display processing to the centralized side.
Why filter processing should be more centralized long-term:
All web browsers (even iPad and newer Android devices) are successfully able to do real time interpolation in my ultralowlag interpolation algorithm (only 1 refresh cycle latency) that I use at www.testufo.com/vrr -- provided the browser is GPU accelerated. The same algorithm I invented, can be used for fixed-framerate to fixed-framerate conversion -- aka standards conversion -- smooth 50fps PAL on 60Hz NTSC -- and smooth 60fps NTSC on 50Hz PAL. Or 100Hz or 120Hz or whatever.
Bitmap Mathematics Now Available In Single-Line JavaScript
All desktop browsers now has the HTML5 extensions necessary, and these capabilities are now generally autodetectable (bitmap mathematics operations where two bitmaps are mathed together in one line of code involving addition, subtraction, AND's, OR's, XOR's, brightness, contrast, saturation, alpha, etc), and I've come up with algorithms that produces really usable mass-benefits (e.g. HLSL simulation without shaders) and niche-benefits (e.g. software BFI). By 2025, even bottom-barrel 10-year-old android devices can do it at 60fps with full-retina-resolution CRT filter emulation, and thus, is no excuse to exclude central-filter-processing from Emularity
Even my VRR emulation algorithm uses simple bitmap mathematics, no shaders, no per-pixel processing -- AFAIK, I think this is the most JavaScript-efficient method of destuttering to fit a fixed-Hz emulator into a random-Hz browser visitor;
Why let downstream emulator modules inefficiently do non-HTML5-optimized filter processing -- when doing it properly JavaScript-optimized in a centralized JavaScript-optimized filter processing layer?
Relevant API References
From just merely these two sets of APIs, I've been able to pull of miracles that formerly required GPU shaders.
For those performance naysayers -- currently, an RTX3080 is capable of approximately 20,000 fullscreen bitmap mathematics operations between 1440p framebuffers (in Windows Chrome, GPU acceleration enabled), and an 5-year-old Intel GPU is capable of about 500 to 800 fullscreen 1080p bitmap mathematics opreations in JavaScript -- sufficiently fast enough for basic CRT filters using prerendered masks (triad of monochrome masks) stored in bitmaps to composite realistic phosphormasks, via a final bitmap-addition operation of the R/G/B channels. It will not do curved filters, but it would accurately emulate rectangular CRTs and its phosphordots as well as MAME HLSL without requiring GPU shaders.
With 100% pure HTML5/JS, emulating the phosphor texture of a SONY Grand Wega accurately would be easy on a 4K display on most recent NVIDIA/AMD GPUs today on any current GPU accelerated web browser (and spectacularly accurate texture if it's an OLED HDTV like a LG 4K HDTV, though even the 48CX is bigger than a 34" Sony Grand WEGA). One needs to do approximately 4-to-7 bitmap math operations cascaded per 60Hz emulator frame (original resolution emulator frames), to get near MAME HLSL-quality full-screen-resolutoin framebuffer. In detecting insufficient performance, it can automatically downrez (1080p CRT filter) until frames goes back up to full emulator framerate, but the full browser capability of a visiting user should be automatically milked, especially when minimum common denominator means 90% of the time full resolution filters can be used.
Most likely, for accurate resolution-adaptive CRT filters, a one-time prerender of precomputed filter bitmaps is needed (~0.1ms to 100ms overhead, depending on system) upon every browser resize event (or zoom/unzoom of fullscreen), but other than that, it will largely autoadapt to the full resolution of the pixel dimensions of the emulator box that Emularity runs in. It's shocking how fast-performance the bitmap mathematics APIs are now in the modern GPU-accelerated browsers because they're just hardcoded-in-drivers shader wrappers in most GPU-accelerated browsers. Driver performance do vary tons (and some `opensource contributions to improving opensource drivers are kinda needed) but, fundamentally, it goes at nearly full memory speed -- and some GPUs can do almost a terabyte per second of bandwidth, which means tens of thousand fullscreen bitmap mathematic operations per second!
In GPU accelerated browsers, an offscreen canvas now behaves just as fast as a 3D texture and the common JavaScript bitmap math operations are now automatically run in GPU optimized parallization, at full GPU memory bandwidth speed (on-GPU, even for bitmap copy-to-copy and mathing).
Extrapolate this 5 years and you'll not be asking questions about tomorrow's smartphones and $200 Chromebook GPUs ability to do real MAME-quality HLSL CRT filters in JavaScript-optimized bitmap mathematics.
Granted, Linux and open source video drivers sometimes have low performance (if you clicked the TestUFO links on Linux, make sure you have GPU acceleration enabled for the animations to work correctly). That said, performance typically can be automatically be detected. And performance is rapidly catching up. It's just a matter of time. I am impressed by the work of the kwin-lowlatency team, by the way.
And shaders can still be accomodated for even more realism later on; since CANVAS layers can be passed to WebGL for shader processing too, if one desired. But one needn't worry about the complexity; the important thing is a slight rearchitecture of Emularity to optionally enable middleman processing.
Besides, self-monitoring of JavaScript performance can also automatically disable CRT filters that don't fit the performance budget of the particular emulator visitor; so we don't even need t worry about waiting for lowest-common-denominator capabilities.... In the first 100ms of Emularity operation, it'll already probably know how fast its own JavaScript and bitmap mathematics is, and autoadapt accordingly, or it can be enabled optionally on a user hotkey (if power user knows they're running a GPU accelerated browser). Brief tests (via TestUFO bitmap mathematic tests), shows even an iPad now has enough power in the mobile GPU for retina-resolution MAME HLSL in pure non-shader bitmap mathematics JavaScript, which is downright impressive; witnessing MAME HLSL in realtime in a web browser.
Architecturing Requirements
This item doesn't specify filtering capabilities, but the eventual ability to add filters. For this item to be closed;
Most filter processing only need current framebuffer, but some temporal filters (e.g. standards converters -- e.g. de-stuttering 50fps on 60Hz displays -- or some advanced BFI algorithms) may need one or two frame lookbehind.
The emulator doesn't need to be aware of any existence of a filtering mechanism, and just only need to pass along emulator framebuffers at regular emulator refresh cycle intervals (just like it is already doing today, already to a CANVAS) -- just that the CANVAS now becomes offscreen / solo bitmap context (it's the same object instancof/typeof!) and is copied for filter processing abilities. This is a unidirectional workflow for the emulator -- no need for Emularity to "communicate back" to the emulator, even during browser resize events.
For simple filters, minimum processing latency will be the time interval of 1 or 2 bitmap copy operations (1/500sec on Intel, to 1/20000sec on RTX 3080), and could perhaps be disabled (provide context to direct Canvas instead of the offscreen Canvas or Bitmap), so there is no mandatory copy performance overhead for the slowest browsers;
Mainstream benefits
Obviously, the more mainstream filters are more useful (better scaling for full screen mode) but the bottom line -- I think it's time to start mentally thinking of architecturing a filterprocess architecture for Emularity -- masterplanning this towards 2025 even if this is impractical right away.
People are visiting websites with ever-better screens (spatial / temporal / color resolution) with ever-better performing browsers with ever-better HTML5 capabilities --
I hope this presents a convincing argument to centralize filter processing / color adjustments / scaling algorithms that are fully HTML5-optimized etc.
Imagine, visiting Internet Archive, and seeing massively better quality emulators when resizing or clicking fullscreen button (more CRTlike scaling, rather than blurryscaling) -- and perhaps even destuttered. There are many benefits of Emularity-side filter processing.