cross platform GUI / 2D software rendering

[nikhedonia]

I've noticed that cosmopolitan supports the creation of gui applications on windows. I'm curious on your thoughts on how you'd go about extending this to the remaining platforms. I'd be happy to make contributions in this direction but I'm not too familliar on how things work on Mac & BSD.

[jart]

Cosmopolitan currently has a razor sharp focus on tui interfaces because it's something we're able to do better than anyone else. I think cross-platform GUIs are great, but I need your support to raise the resources it requires.

The way I'd likely approach the problem is by vendoring something like SDL into the codebase to give you a high-performance blank canvas with audio that works consistently across platforms. I think starting off this way makes the most sense, since games need high-performance graphics and therefore can't be written as web guis like most modern software these days.

Please note that even GUI libraries as simple as SDL usually have a treasure trove of hidden dependencies. Cosmopolitan would need to vendor those too in order to uphold its mission of deterministic hermetically-sealed never-breaking behavior. We accomplish that by not making the assumption that the system did the legwork for us. Therefore it would some research to figure out the best way to vendor SDL dependencies and integrate with window management systems. Here's an example of the research that went into figuring out a way to portably integrate with kernels: libc/sysv/syscalls.sh, libc/sysv/consts.sh, and libc/nt/master.sh.

I can make it happen. I believe it should happen. I don't think 200mb Electron hello world binaries are sustainable. Cosmopolitan can do it better than anyone else, if we raise the resources. So please check out https://github.com/sponsors/jart and tell your friends who might be interested in becoming sponsors.

[jart]

If you want to see some examples of the impact Cosmopolitan is already having in the terminal interface realm, then check out:

• printvideo renders .mpg videos in the terminal
• nesemu1 play nintendo games in the terminal
• apelife conway's game of life tui + gui
• memzoom process/file memory monitor
• blinkenlights pc emulating visualizer debugger
• printimage prints png/jpg files in the terminal
• kilo antirez's famous text editor

One of the reasons why we're able to do terminal interfaces so well, is because the technology is so old (teletypes were invented in the 19th century) that corporations have stopped bothering with the whole engineered incompatibility game and a consensus emerged surrounding the VT100 + termios interface that Cosmopolitan makes easy to use.

[sc0ttj]

Rather than SDL, maybe something like these "single header" and/or "ANSI C" GUI libraries might be easier to use:

• https://github.com/vurtun/nuklear
• https://github.com/nakst/luigi
• https://github.com/floooh/sokol (not a single library, but nice resources)
• https://github.com/nothings/stb (not a single library, but nice resources)

[nicholatian]

The nuklear project is a good thing to start with, however it has been moved and the new location was last updated in December.

luigi does not seem to be portable enough, as it only supports the Win32 and X11 APIs.

sokol seems OK for what it is, however it goes whole hog with GPU acceleration through OS APIs, which is a huge load of maintenance for something like Cosmo to vendor on top of the APIs it already takes care of.

It depends on what @jart wants to do, but I would recommend starting with basic provisions without hardware acceleration, like nuklear, and then slowly graduating to advanced things if desirable. It’s really murky waters given the constraints of Cosmo, so it’s not even clear how much of this stuff she will want to put into project scope. I would recommend caution if expanding scope is part of the prerogative.

[jart]

It took five hundred thousand lines of code to get stdio and sockets working across platforms without dependencies. Doing the same for GUIs would probably take 10x times that, which is roughly the size of the Chromium codebase.

For example, there's an STB style header that does only audio integration which has 63,000 lines of code and it depends on lots of system libraries. Cosmopolitan makes it easy to take "the russians used a pencil" approach of just sending the raw audio samples to an ffplay or sox subprocess:

https://github.com/jart/cosmopolitan/blob/d934f38c997ce43ac92bbe6d3419c4c06c682138/examples/nesemu1.cc#L1704-L1720

I think TUIs deserve more love. Modern terminal emulators make them better than they've ever been. I've recently used ANSI color for ASAN memory dumps. Here's a screenshot of the latest build of Blinkenlights which is an emulator TUI that's hosted in the Cosmopolitan codebase:

Behold the paneling algorithm that makes it possible:

https://github.com/jart/cosmopolitan/blob/d934f38c997ce43ac92bbe6d3419c4c06c682138/tool/build/lib/panel.c#L28-L155

Thanks to Cosmopolitan this emulator works exactly the same on FreeBSD, OpenBSD, Mac, and Windows. Here's a screenshot of it running a Linux ASAN binary on the Windows 10 DOS command prompt:

Cosmopolitan also makes it easy to build offline web applications that can be distributed as a single file web server binary. See https://justine.lol/redbean/index.html which is hosted in the Cosmopolitan codebase.

Those are just some examples of things we're already able to do better than anyone else. That's the best place to focus the 80% of attention: becoming better at the things we're already the best at. GUIs could be fruitful. I love STB style headers and this project has stb_image checked-in to third party. Although if I wanted to pursue GUIs then, based on what I just researched I'd probably try to find a way to build Chromium with Cosmopolitan. I've already borrowed a few things from their codebase, such as their zlib fork. It's astonishingly high quality code.

[nicholatian]

What is the path of least resistance towards getting a no-hwaccel canvas running with the APE polyglot in use? Is this something that needs attention to specifics of the various platforms with fallbacks? If you believe this to be true @jart, I can get to work on that, going as far back as the original VGA spec, or even older if anyone cares. There sure is some novelty in Hercules and composite CGA, if nothing else.

Also, what is the right approach towards having different codepaths depending on system features? In the basic sense, different windowing systems and old VGA-level stuff seems straightforward, but expanding that notion to the graphics APIs is something of a holy grail in this sense. It will be helpful to be able to detect details in a quasi-runtime state, even if that state is only known to the glue code provided by Cosmo, not the application itself.

[jart]

Cosmopolitan can already create a blank canvas on WIN32. If we can do the same thing for X11 then that will cover all our bases. Although on Mac it'd be a little ugly since the user would have to start the X11 program which IIRC comes included with Mac OS.

The way I would probably approach the problem is to build the hello world x11 program from Rosetta Code and then use strace to reverse engineer the protocol. It appears the way it works is you need to open a UNIX domain socket to /tmp/.X11-unix/X0 and then you just send a bunch of binary frames back and forth. Here's what I got on Debian:

socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X0"}, 20) = 0
getpeername(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X0"}, [124->20]) = 0
access("/home/jart/.Xauthority", R_OK)  = 0
openat(AT_FDCWD, "/home/jart/.Xauthority", O_RDONLY) = 4
fstat(4, {st_dev= etc. etc.
read(4, "\1\0\0\6debian\ etc. etc.
read(4, "", 4096)                       = 0
close(4)                                = 0
getsockname(3, {sa_family=AF_UNIX}, [124->2]) = 0
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="l\0\v\0\0\0 etc. etc.", iov_len=12}, {iov_base="", iov_len=0}, 
           {iov_base="MIT-MAGIC-COOKIE-1", iov_len=18}, 
           {iov_base="\0\0", iov_len=2}, {iov_base="\211\32 etc. etc.
recvfrom(3, "\1\0\v\0\0\0\303\t", 8, 0, NULL, NULL) = 8
recvfrom(3, "\240*\267\0\0\0\300\4\377\377\37\0\0\1\0\0\24\0\377\377\1\7\0\0  \10\377\0\0\0\0The X.Org Foundation...
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{iov_base="b\0\5\0\f\0\0\0BIG-REQUESTS", iov_len=20}], 1) = 20
etc. etc.

So yeah I would be pretty happy if we could talk to X11 directly without needing to touch Linux distro shared objects. X11 is a really old protocol that's been around since the 80's so I can't imagine its wire format will be changing anytime soon.

Once we're able to talk to it, we just need to figure out what the magic numbers are to:

1. Get the size of the window
2. Blit an RGB plane onto it

Then what you do with Cosmopolitan is you have if statements ilke this:

if (IsWindows()) {
  mdc = CreateCompatibleDC(ps.hdc);
  mbm = CreateCompatibleBitmap(ps.hdc, r.right, r.bottom);
  BitBlt(ps.hdc, ps.rcPaint.left, ps.rcPaint.top, r.right, r.bottom, mdc, 0, 0, kNtSrccopy);
  // etc. etc. win32 mode
} else {
  fd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
  rc = connect(fd, &(struct sockaddr_un){AF_UNIX, "/tmp/.X11-unix/X0"}, sizeof(struct sockaddr_un);
  if {rc != -1) {
    // etc. etc. x11 mode
  } else {
    for (y = 0; y < yn; y += 2) {
      if (y) printf("\r\n");
      for (x = 0; x < xn; ++x) {
        printf("\033[48;2;%hhu;%hhu;%hhu;38;2;%hhu;%hhu;%hhum▄",
               rgb[y + 0][x][0], rgb[y + 0][x][1], rgb[y + 0][x][2],
               rgb[y + 1][x][0], rgb[y + 1][x][1], rgb[y + 1][x][2]);
    // etc. etc. ansi terminal mode
  }
}

The last part for ANSI terminal mode is what I'd imagine would happen if the program can't figure out how to connect to the X11 server. See https://gist.github.com/jart/7428b2b955dfd6eff7b6d31e00414508 for an elegant explanation of how that's done.

[nicholatian]

If Cosmo supports running from BIOS, doesn’t that necessitate VGA support?

[nikhedonia]

one simple approach on linux is to write directly into the active framebuffer: /dev/fb0. To draw you just write a bytestream where each pixel is represented by one RGBA integer. However that requires root permissions, is completely unmanaged and any application can override your pixels. If one doesn't run any GUI at all, this might be a very pragmatic solution.

http://seenaburns.com/2018/04/04/writing-to-the-framebuffer/

[nicholatian]

Writing to /dev/fb0 is a lot more viable than you say it is. It seems that all is really needed is to ensure the user is part of the video group, and to be outside the X11 instance, on another TTY. The console will overwrite, but only upon program exit anyway, so cleanly exiting is a matter of blanking the buffer out and a call to clear. sudo is not necessary.

[jart]

Framebuffer is totally doable. Cosmopolitan already has support for it too. See PRINTVIDEO.COM which should still be able to play MPEG videos in the framebuffer even though that support was never fully fleshed out, since I normally only use Linux via SSH.

https://github.com/jart/cosmopolitan/blob/e345b42d7847fc8478c1760169fd93bc6f62452d/tool/viz/printvideo.c#L1391-L1463 https://github.com/jart/cosmopolitan/blob/e345b42d7847fc8478c1760169fd93bc6f62452d/tool/viz/lib/writetoframebuffer.c#L22-L41

As for BIOS VGA that's equally simple to do. By default BIOS loads programs in teletype mode. With two lines of assembly e.g. int $0x10 you can put it in VGA mode and then just blit the pixels to a fixed memory address. There are some demo programs in the Cosmopolitan codebase that do this for CGA which is almost the same thing as VGA. Blinkenlights can even emulate it.

make -j8 MODE=tiny o/tiny/tool/build/blinkenlights.com o/tiny/tool/build/emubin/spiral.bin
o/tiny/tool/build/blinkenlights.com -rt o/tiny/tool/build/emubin/spiral.bin

[hippi777]

hi all! :)

i just came to suggest some nice gems to check them out:

• tekui: it is designed to be embedded on the 1st place, it has an isolated driver layer with linux fb, raw fb, direct fb, x, and w*ndos support included, it is c99, and it knows kinda much all i want for graphics, even if i wanna redesign it in the future to align it to a currently private project. see the last few messages on its mailing list for more info, and their website http://tekui.neoscientists.org/screenshots.html
• arcan: this has really nice features, however its not really mature, and its codebase is kinda ugly (ymmv) https://arcan-fe.com/about/
• tiny core linux: it has a thinned-down x that is just enough to run fltk
• temple os: it has a hypertext tui interface, while it has 3d graphics, but it is 16 colors only (Terry misunderstood God, the restriction of having only 16 colors was only for him, just to keep the code base simple, and to keep him away from wasting too much time on playing... btw, worry none, he went to heaven after all, cuz of his mission! :) ) fwiw, it supports fat, and there was a decompressor around, that was written to work on linux, so it should be possible to extract the plaintext codes

btw, u r about to go for x, but it can be considered to be depricated besides wayland, even if it will stay for a long while...

(ps: im only around to learn some nice hacks, and cuz the c lib benchmarks are impressive, but multi platform stuffs are actually out of my interest. i might have a high latency, cuz of an extreme amount of todo...)

bests, have fun! :)

[jart]

I'm closing this because Cosmopolitan intends to cater to the needs of people building online production services who want standard interfaces like serial / sockets / stdio / termios and tools like Wayland / X11 / Unity / TempleOS are the last thing they'd want going near production instances, although of the four TempleOS would probably be the least impactful since it uses VGA and SGABIOS is able to VGA displays back into SERIAL UART which effectively makes it headless once more. That said you can expect a change real soon where I'll be adding a linux kernel style negative memory managemer to your αcτµαlly pδrταblε εxεcµταblεs so that they'll be able to call C standard library functions like malloc() when you boot them on BIOS or UEFI.

[hippi777]

i didnt mean for those to be used as they are, on the 1st place, but basically as good resources... otherwise feel free to do whatever u want! :)

[paulreimer]

Thou shalt not vendor TempleOS pls

[niutech]

Wouldn't it be possible to create a cross-platform GUI using e.g. LVGL or Nuklear on top of MiniFB or pxCore?

Or use GLFW, since OpenGL is already bundled in every major OS?

[jacereda]

I do have a quick and dirty proof of concept (based on microui) for something that could work but isn't ideal at https://github.com/jacereda/rui/

rui.c is a microui renderer that sends quads via UDP to a local or remote program that will just render them using OpenGL (ruiview.c). The viewer sends back events to the microui side. The viewer should run on Windows/Mac/Unix.

So, this is sort of poor man's Electron.

What would be really nice is being able to embed the different platform versions of my canvas library (glcv) in the executable. It's really minimal and I guess the overhead would be a few KB. If dlopen could resolve X11/OpenGL symbols we could have cross-platform OpenGL executables.

I think something like what https://github.com/jart/cosmopolitan/issues/278 describes could work for the X11/OpenGL use case, what do you think?

[jart]

All you have to do is put the platform local binaries inside the zip structure of the binary. You have to build the binary with a STATIC_YOINK("zip_uri_support"); statement. Then you can say if (IsWindows()) open("/zip/opengl-driver.exe", O_RDONLY); etc. etc.to move the binary to a local file system folder. Then you can call pipe(), fork(), and exec() to communicate with it as a subprocess.

[jacereda]

I have been exploring another approach and I think I can get cross-platform OpenGL rendering on the same process (no IPC). I've only tested the Linux side, I'll try to finish the Windows side to have a complete POC.

[jacereda]

[pkulchenko]

@jacereda, nice! What are the dependencies here?

[jacereda]

It's just linking statically against a modified https://github.com/jacereda/glcv and loading at runtime opengl32.dll and dwmapi.dll (for the transparency support). On Linux, libGL, libX11, libXrender and libXcursor.

[jacereda]

And to lookup OpenGL functions I'm using a slightly modified Galogen file.

The demo itself is from https://github.com/rxi/microui

[jacereda]

I need some time to cleanup the mess and publish the whole thing in a not-so-embarrassing state.

[jart]

Reopening because seeing is believing. @jacereda if you're willing to lead the Cosmopolitan GUI effort then I can support you. Your work is welcome here and would be appreciated by many people. For example @lemaitre recently began leading our Cosmopolitan Threads effort in #301 and #282 which is another exciting development, but still a work in progress.

[jart]

It's also worth mentioning that, thanks to @fabriziobertocci, we have support for UNIX domain sockets. Therefore it should now be possible to hack the X11 protocol to get a canvas. It'd be super cool if we could just do that without the runtime dependencies. Dependencies feel more real when you communicate with them using binary. It would also likely ensure it works on BSDs and probably XNU too since last time I checked Apple ships an X11 server that can be started. But you almost certainly can't rely on something like a libx11.so file being present somewhere on the file system. It's probably just the X11 wire format that all agree upon. In which case, all we need to do is figure out where the UNIX socket file is likely to be stored on all these systems.

[jacereda]

For Mac I would just use native cocoa. The X11 server is a separate install and most people don't use it.

As for talking directly to the X11 server, I would just stick to dlopening libX11. At some point it would be good to support Wayland, implementing the X11 protocol sounds like beating a dead horse.

Also, graphics is the beginning, we'll also need audio and something like https://github.com/andrewrk/libsoundio would need to be ported. Avoiding runtime dependencies sounds cool, but I don't know how well it will scale.

[jart]

Avoiding runtime dependencies is cool because we're the first people who successfully managed to do it. It is scalable because O(1 statically-linked version) is less costly than O(n dlopen versions m operating systems o display managers)

What if we just do Windows? Having OpenGL GUIs on Windows will certainly be progress over the status quo. Then we write a simple driver that can automatically display the OpenGL canvas inside the terminal, when the executable is run on other systems. We can then immediately lay claim to having a solution for building GUIs that run on seven operating systems. If people love it then we can raise the resources to do all the super expensive kind of portability like dlopen() that would enable us to offer a native look and feel on additional platforms. Thoughts?

[niutech]

@jart Here is the discussion about outputting OpenGL without X, where they mention DirectFBGL and OSMesa.

[Kokokokoka]

also https://arcan-fe.com design can help too.

[rswinkle]

Sorry to jump in here, I just stumbled across Cosmopolitan today and immediately looked for the status of graphical support. What's already shown ITT is amazing but I also liked the idea of integrating SDL2 as mentioned much earlier.

Whatever ends up happening, I hope a way of just blitting full color framebuffers to the screen becomes possible. It would enable me to use PortableGL to create a truly cross platform portable "OpenGL" program easily rather than having to deal with cross compiling or VMs.

Thanks for such a cool project and making life more interesting and fun for us C programmers.

EDIT: broken link

[ghaerr]

Hello all, I thought to jump in here with my thoughts, since there seems to be some ongoing interest in GUI (vs TUI) for Cosmopolitan. I'm new here, please excuse me if I'm commenting on something already known to the reader.

With regards to GUI it's sometimes helpful to make a distinction between high-level and low-level issues. When many people talk at a higher level of "wanting a GUI", what most mean is they have an already written program (or are using a known API) and want that program available on thus-and-such a platform. Few are willing to go to the trouble of rewriting an application to a new GUI API.

Of course, the lower-level APIs such as MSFT win32 and X11 Xlib come to mind, but, like network stacks, the implementation of not just the API, but the integration into a graphics stack is important, as for example requirements existing where a hardware GL renderer may want to be switched with a slower software renderer. There's also the problem that many applications use higher level GUI libraries written on top of the lower level win32/X11 APIs, and those libraries need to be ported as well, complicated the porting.

So a good part of the "what GUI for Cosmopolitan" discussion would have to include discussions of larger GUI libraries. For the larger and more modern GUI libraries, the "vendoring" problem can be formidable, and I'd have to agree with @jart that advanced TUIs can work quite well for lots of requirements, foregoing the vast vendoring and maintenance job of full-blown modern UIs.

However, for smaller applications (which are more my interest), there are some paths that could allow integration of very useful graphics/GUI capabilities into Cosmopolitan with no dependencies. With a known low-level handoff, such as frame buffer or perhaps OpenGL (or both), this could be accomplished because all of the higher-level graphics are actually just algorithmically computed (fast) byte-buffer block (blit) and text/draw routines, which all operate in memory with no system calls, at least on a software implementation.

As an example, at the Microwindows Project (disclosure: I am the maintainer), this very small system implements both the win32 and X11-like nano-X APIs, on top of a small graphics engine which implements software-only text and drawing functions, which then sit on top of a low-level driver interface, such as frame buffer, SDL, X11, WASM browser canvas, etc. The entire system is capable of running on bare hardware, with only desired dependencies, which are usually various image decoders.

Other APIs, such as RXI's microui (suggested by @jacereda), a tiny immediate-mode UI, or nuklear, a larger immediate-mode UI, sit on top of Microwindows and are thus redirected onto framebuffer, SDL or X11 lower-levels. Here's the standard microui demo running on Microwindows (in this case on top of SDL on macOS):

The cool thing about microui is that the entire draw code for the above displayed output is through only two graphics routines: text and fillrect only (!!). These could be implemented in software with just a small frame buffer engine and blitter, or, as @jacereda suggested above, switched out to a hardware or external OpenGL renderer.

While not suggesting Microwindows or microui for a starting point, it seems that, should there be real interest, implementing an initial software-only architecture that allows for a variety of mid-level APIs (win32, X11, etc) on top of perhaps frame buffer initially, but with the capability of switching out pieces of the graphics stack (or even integrating within the TUI) could make sense.

For the frame buffer case, output to Linux would be available through just the open, mmap and ioctl system calls. For macOS, a vector through Cocoa would likely need to happen, unless SDL were used. SDL brings about its own much larger set of vendoring issues, but provides keyboard and mouse handling, along with the associated event handling; these last three items, event handling, mouse and keyboard, are another large complication on a per-platform basis, something that's not been discussed much yet on this thread.

Should it come time to vendor larger pieces of graphics stacks to Cosmopolitan, I have additional thoughts on porting issues.

Thank you!

[ghaerr]

Hello,

Just for fun, I took Cosmopolitan's "life" program, which has dual TUI and GUI (win32) user interfaces and played around with getting the GUI code to run on top of Microwindows, which implements the win32 GDI API entirely in software. The idea was just to see how slow such a thing might be, and what efforts it might take to port it.

Here is the result, running on macOS on SDL (for the time being):

https://user-images.githubusercontent.com/11985637/161162845-fd83693b-15bf-42a2-90f3-be279dcd55eb.mov

Note - this isn't running within Cosmopolitan (yet); an idea was that it might be feasible, or at least fun, to execute some of the Cosmopolitan win32 API calls (when not running on Windows) in software within Cosmopolitan, with the whole thing output to frame buffer. However, that frame buffer could be virtual, rather than a Linux kernel frame buffer.

While the above demo produces bits in a frame buffer that is sent to SDL, the next step would be to output to a virtual frame buffer with access through Microwindows' frame buffer emulator. That is, the actual graphics bits would be calculated from the application on down through the win32 API and graphics engine to a virtual frame buffer, where the frame buffer would actually be a shared mmaped file, while the access to the frame buffer, possibly from another machine, would be through a frame buffer emulator (FBE). This would allow win32 graphics applications running on Linux or macOS accessed through an FBE on an X11 client, or output directly to a hardware Linux frameuffer, for instance.

The benefit of all this would be that, if ported to Cosmopolitan, any graphics would be self-contained within the APE executable within the OS support vector, with accessibility through a shared graphics frame buffer in a multitude of ways.

[jart]

Are you proposing that we polyfill the win32 gdi api for other platforms? I must admit the thought has crossed my mind...

[ghaerr]

Are you proposing that we polyfill the win32 gdi api for other platforms?

Possibly, but the idea's not without its problems. In staying with Cosmopolitan's stated principles, everything must be completely contained. That could be done, but it gets more difficult because of win32's built-in controls, all accessible through predefined window classes. That's potentially a lot of code, and and needs to stay small, IMO. However, all the graphics code would be platform-independent from the start, with only the frame buffer or FB emulator becoming platform dependent.

I've been deep in your code learning about the repo and all that it contains, pretty amazing work. Now, I'm playing with various graphics ideas that could be somewhat easily "hermitically sealed" within Cosmopolitan, with win32 being the first. It has the benefit of being widely used, but disadvantage of possibly becoming too large. Another thought is the nuklear immediate-mode UI, which has some wide use. It's considerably lighter weight and has an updated look and feel.

I have Nuklear now running direct to frame buffer emulator using an X11 frame buffer emulator to access it. This means it could be entirely encapsulated within Cosmopolitan fairly easily, and allow users to create their own UIs.

Here it is running on framebuffer emulator on macOS, using an X11 client to access it, without flicker. (I'll create a movie of it if you like, which shows off much nicer):

If the frame buffer emulator were rewritten to use SDL or something else rather than X11, and/or just move bits directly to a hardware Linux frame buffer, that might work well.

[ghaerr]

Here is a nuklear immediate-mode UI movie demo using virtual frame buffer. This means all graphics programming could be completely encapsulated in a platform-independent way for each graphics application, whose authors would use the nuklear API to write their applications on top of Cosmopolitan.

In this case, which is a bit different than win32, each nuklear application would be a Cosmopolitan binary, issuing small "nano-X" drawing commands over a named pipe to a nano-X server, which would be another Cosmopolitan binary. Since the applications are not doing any actual drawing, they remain quite small (kind of like X11 apps).

The nano-X server does all the drawing, and allows multiple nuklear applications to attach simultaneously, acting like a window manager. However, the NX server draws only to a virtual frame buffer, so it could be quite platform independent. It gets its mouse and keyboard input via a fixed tiny protocol from the frame buffer emulator.

The frame buffer emulator, here running as an X11 client on macOS, would probably not be a Cosmopolitan binary. It accesses the virtual frame buffer via a shared mmap file, and the keyboard/mouse via two named pipes. It all sounds a bit complicated, but the important point is the ability to keep the graphics component of the APE binaries very small and platform independent.

Notice there is no flicker at all, even though all frame buffer bits are being rapidly interpreted (only when changed) by the frame buffer emulator.

https://user-images.githubusercontent.com/11985637/161183889-3956be6d-f662-4266-a46c-1f82b2343359.mov

[jart]

Possibly, but the idea's not without its problems. In staying with Cosmopolitan's stated principles, everything must be completely contained

A lot of effort has gone into making everything completely contained. However I don't want hard stances to block progress on other things. For instance, I've had some local work in flight for a while to hopefully unblock the OpenBSD GUI work that's been happening in this thread. Since I think people want to see things happening above all (great screencasts by the way!) even though the most benefit is going to come from working towards a state where the user take full advantage of the self-definedness. That way you don't get things like distros changing DSOs ABIs breaking your apps. Not having to deal with ABIs has been a huge productivity booster for me, as anyone who's poked around the repo has probably noticed :-)

[jart]

One thing I'll note though is that, even if we need to link DSOs on some platform to support GUIs (like technically we link DSOs on WIN32 because there's no other way, but MS ABIs are super stable!) then it's absolutely never going to compromise the default link paths for platforms like Linux. Not ever.

[niutech]

@ghaerr Great effort, thank you! Do you have any demo Cosmopolitan binary (APE) of Nuklear app to test in Windows or Linux with fbe?

[ghaerr]

Hello @niutech,

I am sorry, I don't yet have an APE binary of a Nuklear app to test with. I am still a couple steps away from that, and am gauging interest, to see whether this approach makes sense, or whether another method would work better.

Also, in order for this to work with keyboard and mouse, a modified version of fbe would have to be used, which is currently tied to the nano-X window system for mouse and keyboard I/O. Both FBE versions require X11 for display output, and it would be nice to port FBE to SDL, for broader compatibility. So things are still a bit overly complicated.

The next steps to getting a reasonable APE demo would require compiling Nuklear, as well as portions of the nano-X API and its underlying graphics engine with Cosmopolitan tools and header files. While the graphics sources are quite portable, I would like to do this without having to add them to the Cosmopolitan source tree, and would like to use standard header files. I am currently investigating how to best do this, while getting Cosmopolitan itself compiling well on macOS.

Thank you!

[ghaerr]

Hello @niutech,

I (finally) have a Cosmopolitan APE binary demo of the Nuklear NodeEdit application I uploaded a screencast and screenshot of above. This is a completely self-contained binary which computes all graphics completely internally, as well as handles keyboard and mouse input: demo.com.zip

At this time, demo.com binary talks to an X11 based frame buffer emulator via some two shared memory files created in /tmp, as well a couple of named pipes in /tmp for keyboard and mouse input. These are created and handled automatically though:

-rw-r--r--  1 greg  wheel      512 Apr 12 15:39 /private/tmp/fbe-cmap
-rw-r--r--  1 greg  wheel  3145728 Apr 12 15:39 /private/tmp/fbe-framebuffer
prw-r--r--  1 greg  wheel        0 Apr 12 15:41 /private/tmp/fbe-keyboard
prw-r--r--  1 greg  wheel        0 Apr 12 19:46 /private/tmp/fbe-mouse

You will have to use my enhanced FBE (frame buffer emulator), which comes from picoTK as yours did, but has enhancements to pass mouse and keyboard input over to the APE binary: fbe.c.zip You will have to compile this binary on your host Linux or macOS platform using:

gcc fbe.c -o fbe -lX11

Then, run fbe & and demo.com and you should be operational! demo.com should produce the following output, and allow you to operate the mouse and keyboard for running the nuklear UI application:

Note: all of this was created on macOS, which is not (yet) standardized on Cosmopolitan, but I hope will be shortly. @jart and I have a few more things to work out. Thus, this has not been tested on Linux! With a bit of work, it should be fairly easy to use Microwindows/nano-X to port any other Nuklear-based UI programs fairly easily to Cosmopolitan, without having to retrofit them with #include "cosmopolitan.h" or change from using the normal standard header files. That is, they would be ported without source code changes.

Thank you!

[niutech]

Thank you very much @ghaerr! I have compiled and run fbe under WSL & Xming and I can see its window, but when I run demo.com I get these errors:

Cannot initialise keyboard
Unable to open graphics

even though these files exist:

/tmp/fbe-framebuffer
/tmp/fbe-cmap
/tmp/fbe-mouse
/tmp/fbe-keyboard

I will try later on full Linux desktop. But how to run in on Windows or WSL?

[ghaerr]

Hello @niutech,

Cannot initialise keyboard Unable to open graphics

The first error above is the graphics engine reporting failure on open("/tmp/fbe-keyboard, O_RD_ONLY|O_NONBLOCK), which then displays "Unable to open graphics" and exits.

The two files /tmp/fbe-keyboard (and /tmp/fbe-mouse) are named pipes created by FBE using mkfifo("/tmp/fbe-keyboard, 0666). Since the open failed, it would seem there may be an issue as to whether named pipes are supported in Windows and/or Cosmopolitan. I suppose it is possible there may be other permissions/access issues related to Windows.

@jart may have some input on this, I haven't looked deeply into the named pipe translation, there have been a lot of Cosmopolitan enhancements for The New Technology (Windows) lately and my repo could be out of date. (I plan to fix that after getting macOS builds fully working shortly).

[EDIT: Now that I think of it, my SUPPORT_VECTOR on macOS builds isn't yet set to include NT, but demo.com was built using a Cosmopolitan-provided v1.0 cosmopolitan.a which should have NT support included. Thus demo.com only has v1.0 NT support built in.]

Thank you!

[ghaerr]

I have now completed the work of getting nano-X building all the Nuklear and most nano-X demos for Cosmopolitan, including the nano-X server. This means that the file sizes for each of the graphical programs are smaller, as they don't do any actual drawing. Instead, each .com binary talks with the nano-X server via a UNIX socket, and the nano-X.com server talks with FBE via named pipes and a shared mmap file (described above).

If interested, one can compile up and play with this themselves, by doing the following:

git clone https://github.com/ghaerr/microwindows.git
cd microwindows/src
cp Configs/config.cosmo config
(edit config and set the path of COSMO{INCLUDE,LIB,APE,CRT} to your Cosmopolitan installation or downloaded amalgamated library)
(may have to set X11{LIB,HDR}LOCATION as X11 required for final bin/fbe frame buffer emulator build)
make

This will produce APE files in microwindows/src/bin, including bin/nano-X.com (the nano-X server) and demo-nuklear-*.com files, as well as the host bin/fbe (frame buffer emulator) .

Executing the following will produce the demo shown below (yes, nano-X roaches hiding under windows thrown in for fun):

bin/fbe &
sleep 3
bin/nano-X.com -p &
sleep 2
bin/nxeyes.com &
sleep 1
bin/nxroach.com &
sleep 1
bin/demo-nuklear-calculator.com &
bin/demo-nuklear-node_editor.com &

https://user-images.githubusercontent.com/11985637/163303468-87b287de-a159-4308-a6e0-cdc620981d14.mov

Any Nuklear graphical .com program started will automatically connect to the server, and work with the keyboard and mouse. It will exit when the close box is clicked.

It would take some work, but FBE could be rewritten to use SDL or something more modern than X11, providing better portability for some systems.

[niutech]

@ghaerr I can confirm your demo.com works in fbe in Ubuntu 20.04. Good job, thank you!

Now let's make it work in Windows.

[ghaerr]

Hello @niutech,

Glad it's working on Linux, thanks for the testing!

Now let's make it work in Windows.

Now that you can compile up this demo directly yourself from the nano-X repo as described above, I suggest you do that linking with cosmopolitan.a from its latest version. You may find it all works, as my Cosmo repo is a bit outdated for the time being.

Should it not work, perhaps @jart or @pkulchenko could tell how to turn on system call tracing so that we can determine what the problem might be. I am guessing that it is likely something to do with either UNIX sockets or named pipes on Windows, if a recompilation with the latest repo still fails. The vast majority of the nano-X code is completely portable across operating systems.

Thank you!

[niutech]

@jacereda's cosmogfx demo works in Windows 10. Could anybody confirm it works in other OS-es? If yes, it is by far the simplest way to run cross-platform GUI apps - a single 200KB APE file!

[ritschwumm]

@niutech looking good on ubuntu 22.04

[mingodad]

It segfaults on Ubuntu 18.04.

[jacereda]

It segfaults on Ubuntu 18.04.

It should probably be due to different glibc versions. The binary embeds prebuilt ELF files for this program:

https://github.com/jacereda/cosmogfx/blob/main/helper.c

Detecting glibc version and loading the appropriate helper binary could be an option, but I don't have the time/energy to write that.