Add embedded web server for monitoring, management, websocket based IO connections

[dmcnaugh]

Priority: medium Difficulty: difficult

I already have a decent library of frontend components that run in the browser (all native HTML/CSS/JavaScript)

• simulated GUI desktop, including the following virtual/emulated devices (some that I just made up)
• TTY: - VT100/ansi compatible character mode serial terminal, can be windowed or zoomed to full-screen (over websockets)
• CRT: - complete implementation of the IMSAI VIO (over websockets: mirrors the 2KB display RAM and does all rendering in the browser)
• LPT: - complete implementation of an 80-column dot-matrix/line printer with selectable paper-types - ready to print on your real printer (dumb text printer, no Esc codes, over websockets)
• X:DSK: & LIB: - graphical disk manager for the 4 floppy drives A-D, including eject, insert (drag-and-drop) from a disk library (.../machine/disks/library/), download disk image, upload disk image, delete disk image (uses RESTful web service interface)
• CPA: - IMSAI CP-A front panel (only the 6 control switches & 4 status LEDs on the RHS, over websockets)
• SYS: - system status window, shows details of the simulation (host?), enables remote reboot
• CFG: - configuration file editor, enables config files (.../machine/conf) to be edited and saved

You can see what they look like in the attached screen-shots.

These all work against my port of the imsaisim to the esp32 micro-controller platform. When the browser is made full-screen you would't know you were in a browser, it just feels like a desktop GUI.

Unfortunately the embedded webserver I am using in that port is specific to the platform and not available for OSX/Linux/Win. But if a suitable embeddable webserver can be found and integrated with the z80pack then all the frontend code it ready to go.

[udo-munk]

Very cool.

[udo-munk]

Open sourced and MIT license, server source looks good. https://github.com/jeremyhahn/cwebsocket/tree/master/src

[dmcnaugh]

I think I'll try out civetweb https://github.com/civetweb/civetweb It is/has:

• Open source MIT license
• Actively maintained
• Static file serving
• Websockets
• File uploads
• CGI support

and it might be an alternative for me to also use on the ESP32, if its not too hard to port, allowing me not to have to maintain 2 versions of the code.

I'll get a proof-of-concept going with the SIO over websockets to test the viability.

[udo-munk]

Would be great to find one that can be used in all machines.

[dmcnaugh]

civetweb looks like it is cross-platform for Win/Linux/OSX. So far it’s been very easy to work with and I have static file sharing, websockets and cgi figured out. Still have to try setting up for a file upload.

My next problem is queues (fifo). I had hoped to use m_queue but it isn’t available on OSX. My needs are simple so I am going to write my own using an array and a p_threads mutex for thread safety. This solution should also be portable cross-platform.

The final test will be to see if I can port civetweb to the ESP32. If not then maybe I can do an abstraction layer to minimise the code differences.

[udo-munk]

UNIX ipc message queues won't do? These are defined by POSIX and available in all UNIX systems, also OSX. Not sure about Windows/Cygwin, but probably also mapped to the Windows internal message queues.

[dmcnaugh]

Thanks for your suggestion. I am now using sys/msg.h successfully. I only hope I can always close the queues correctly as they don’t close automatically when the process ends. During testing I ran out of available queues and had to reboot. I’ve got TTY: LPT: and a very simplified SYS: working, along with stataically serving all the required files. None if this is refined or ready to release, but it’s not bad for a few days of effort.

[EtchedPixels]

You can use ipcrm to clean up. They are horribly messy things, buggy in a lot of platforms and not always enabled (they are XSI not base). There is a reason everyone has been trying to deprecate them for 20 years 8)

To quote the standard "Application developers who need to use IPC should design their applications so that modules using the IPC routines described in XSI Interprocess Communication can be easily modified to use the alternative interfaces."

I always use pipes or sockets instead. Pipes guarantee that writes below a certain size are atomic so a single write of an object with a length will never get interleaved with another threads write.

[udo-munk]

20 years ago trivial things like IPC just was working 8) Can't be depricated because major products like Oracle and IBM databases, X11 (shared memory), make massive use of IPC, just works.

Of course one can use pipes or UNIX domain sockets, these just come with different problems, I don't know what might work better in this case.

We can add atexit() to clean things up. I was thinking about doing this too for the terminals left in raw mode, when the program runs into one of the few exit(1) for really fatal errors. So far I didn't implement this, because end users won't see an exit(1) (hopefully) and programers know how to clean up messy terminals, ipc etc.

[EtchedPixels]

It's mostly deprecated - X11 Xshm extension still uses it but not the big databases any more. Xshm is on it's way out and shm IPC_RMID can be done when you create an object so it dies on last reference, whereas msg queues IPC_RMID kills it immediately.

atexit() only helps so much because you can't exit() from a signal handler, it doesn't run on uncaught exceptions and _exit() doesn't invoke atexit methods.

[udo-munk]

Hm, I still have to configure IPC kernel parameters for running Oracle or db2.

True, atexit() is not perfect and only helps so much, but helps to avoid all cases where exit() is used for fatal problems. Programs with uncaught signals and exeptions have a design problem, that needs to be addressed in the design and not in some methode used.

[dmcnaugh]

OK, using atexit() for the best chance of closing the queues.

Most things are working now (TTY: CRT: LPT: X:DSK: LIB: SYS: CFG:, see OP #38) just have the CPA: to do.

Its really helped me clean up how the web UI integrates/hooks into the existing code.

On the other hand, the code I have hammered together behind the webUI is currently a very ugly hack of example code, bits back-ported from my ESP32 port (using a different web server library) and lacks good error checking.

[udo-munk]

Sounds good. With regards to the queues, if I remember correct it is possible to figure if a queue still exists and reuse it, without creating a new one. Have to re-read some of the old Stevens books about IPC and can take a look at it.

A suggestion what we can do: You put what you have into dev and let everyone play and maybe improve on it. Before the next release is due I'll let you know and freeze dev. You decide if the code is good for release or not. If not I take it out of dev, create release, create new dev and add the 'ugly' code again, for further working on it.

[dmcnaugh]

Let me know about the queue management if you find anything. It seems to be working reliably now without "losing" any queues. They are being created IPC_PRIVATE if that matters.

Regarding how to release/introduce the code:

• First, I am going to isolate changes that are useful in dev regardless of the webserver/websockets functionality. -- Issue #45, PR #46 is the first example of what I mean - a logging library to use anywhere -- Next will be a diskmanager module, that works independently of the webserver stuff, that can used used by both the imsai-fif and the tarbell_fdc and extended to the others with a bit of work. -- the diskmanager can be selectively included with a HAS_DISKMANAGER macro in sim.h
• Next will be updates to include in dev that provide the integration/hooks back into the webserver -- effected modules include: imsai-sio2, imsai-vio and iosim for TTY:, CRT: and LPT: respectively -- these will continue to work normally without the macro HAS_NETSERVER set in sim.h
• Next will be the webserver/websockets server-side code that is embedded in imsaisim -- this code is improving, and will be easier to introduce with the above code changes already done -- I still need to decide how to bundle civetweb. Its MIT licensed so that's not a problem. Just have to choose between making it a submodule, at the same level in the project as frontpanel is now. Or whether to just bundle it as civitweb.h and civetweb.a (a library that I will configure and compile, then include). -- I also need to figure out how I will bundle and include the client-side HTML/CSS/JavaScript and images.

So a lot to do, but the ball is now rolling.

[udo-munk]

I'll take a look at the queue management and let you know.

Logging library is good too, with growning number of modules needed sometime anyway.

A diskmanager is great, I never found the time to write one, but it will show why I chooses the disk layout as is using a disk library. Love the GUI with the disk filing cabinet icon :)

Just a lib won't do, it runs on too many platforms, so we need sources. Same as with the frontpanel library we well take the current sources and put them into z80pack/webfrontent/src and your content into z80pack/webfrontend/content or so. We always have working server sources then, regardless how the software will be modified, and everyone is free to play with newer versions. If newer version don't work anymore the problem can be ignored or fixed.

With the web contents you should support the same methode as with the other config files, disk library, etc.. First look at ./webfrontend/content, if there use that, else look at /usr/local/share/imsaisim/content or so. This allows to run the extracted software right from it's directory without the need to install anyhing, as well as building packages that install the whole stuff under /usr/local or /opt or whatever.

[udo-munk]

Got it working even on Windows/Cygwin, so very portable everything. It looks awesome, even better than the screenshots above :) Will commit a buch of changes tonite for building ...

[udo-munk]

Ops, not really working with Cygwin:

msgget(): Function not implemented I (8485) netsrv: WS CLIENT CONNECTED to LPT I (9515) netsrv: WS CLIENT CLOSED LPT msgget(): Function not implemented I (10170) netsrv: WS CLIENT CONNECTED to SIO1

[udo-munk]

To enable IPC on Cygwin/Windows: http://cygwin.com/cygwin-ug-net/using-cygserver.html Then it works.

[udo-munk]

With edaf4d5 I'm now able to build with and without web server under OSX, Linux and Cygwin. I used a Debian 8 for Linux testing, this is relatively conservative, so should do OK with other distributions too.

[dmcnaugh]

Just about ready to push a new build of the web UI (includes 1 important server side addition). Here's a preview: Notes:

• 2 new icons: update for CRT:, added for DZLR:
• added DZLR: device that implements the Cromemco Dazzler (only tested with kscope.com, need some programs to test other modes/resolutions)
• made TTY:, VIO:, DZLR: able to go full-screen with the addition of a widget on the window title-bar and the use of the ALT-Escape key combination
• added ability to take screen-shots of the VIO: and DZLR: with the use of a widget on the window title-bar
• enabled the web UI to be run on a browser on any computer that can reach the host machine running imsaisim across the network
• radically reduced the file sizes for the web UI code and assets (HTML/CSS/JavaScipt/images/fonts) resulting in sub-1-second load times of the web UI

If its not already understood, the web UI is developed exclusively to work with the latest version of the Google Chrome browser (Official Build). If it works with other browsers that's fortunate but not tested. Its likely that most features are portable.

Here are some screen shots of the VIO: and DZLR: made with the new screen-shot feature.

[udo-munk]

Verry cool.

In the Cromemco disk library you'll find a disk 'dazzler_graphics.dsk', the original demo disk from Cromemco. The programs on it were written for CDOS, most won't run under CP/M. However, there is a program gdemo with working graphics output, terminal input won't work because of the different BIOS calls. This program demonstrates most or all video modes. It requires a Z80, as almost all software from Cromemco, so use emulation with -z.

[dmcnaugh]

All tested (and fixed). The gdemo program has been very helpful and entertaining. I could spend hours watching the Dazzler run the demo in full screen on my 1440p monitor. And just for fun I was able to Chromecast from the browser to my 4K TV in the living room 8-) The server side code is nor ready in PR #56. The new web UI build will follow shortly.

An extra feature made it in for the DZLR: device:

• ability to change aspect-ratio of the Dazzler display between 1:1 (square pixels), 4:3 (CRT TV), 8:7 (not common, but the closest to the "real thing" I could measure off a YouTube video of an original Cromemco Dazzler working https://www.youtube.com/watch?v=2tDbn1N8EWI, so I've made this the default), with the use of a widget on the window title-bar
• this chosen aspect-ratio is preserved in full-screen mode
• but not when a screen-capture is taken, this is always 512x512 square pixels

[dmcnaugh]

The majority of features even work on an iPad browser (Chrome). So far I've only noticed problems with full screen

• not working for the DZLR:
• working on the TTY:
• broken, half working on the CRT:
• you have to reload the page to get out of any full screen mode

P.S. The yellow, chevron widget in the TTY: is specifically for use on the iPad, it is a Ctrl- key lock.

[udo-munk]

It is awsome, great work.

The Dazzler demo software was published by Cromemco in 1978, so 4 decade ago. Fascinating that some people still love to watch these demos. And yes, I did run it for hours too, after I got the stuff working :)

Create a YouTube video from running the Web desktop, Harry Garland will love it.

[udo-munk]

IMSAI 8080esp kit is shipping, so implementation completed.