[Question] How does clog work conceptually? And which kinds of apps would/wouldn't be a good fit for clog?

[solarmist]

I'm new to Common Lisp and just found Clog recently. It's really interesting and impressive, but conceptually I don't really understand it.

I get the idea of treating a browser window like a screen, but if I left it at that clog couldn't be anything be used for anything more than pet projects, intranet applications or streaming applications, but not for serving thousands of users from a single instance. I expect most people aren't trying to build the next Netflix on the web, but would like more powerful GUI abstractions.

If clog is meant as a pure GUI (client/server) framework then that should be mentioned somewhere. From the docs it's really difficult to tell if it would be appropriate for most web apps; i.e. small infrequent exchanges of data to over an unreliable connection.

I don't want to give up all of the things that web pages give us. Ideally, I want the best of both worlds, rich dynamic interactions, local caching of data, and robustness against network connectivity issues. The other impression it gives me is one of tight client/server relationship (almost like an ssh session). In that case it seems like it would severely limit the number of clients it can service at a single time.

So far documentation has been my biggest gripe with the lisp community. Clog is far better than most libraries I've seen. You have a fair bit of documentation, but it is not written for beginners or people coming from more common ways of writing web code.

From the README

The key to CLOG is the relationship it forms with a Browser window or Browser control compiled to native code. CLOG uses websockets for communications and the browser to render a GUI that maintains an active soft realtime connection. For most CLOG applications all programming logic, events and decisions are done on the server which can be local, or remote over the web.

From the user manual

The Common Lisp Omnificent GUI, CLOG for short, uses web technology to produce graphical user interfaces for applications locally or remotely. CLOG can take the place, or work alongside, most cross-platform GUI frameworks and website frameworks. The CLOG package starts up the connectivity to the browser or other websocket client (often a browser embedded in a native template application.)

These explanations are a great to get people excited about it, but are insufficient once you want to jump in with an idea or project.

I.e. So, you go to the URL and get some HTML, CSS, jquery, and boot.js, which connect a websocket back to the server. Ok, then what? If I have a drag-gable window; am I streaming mouse positions to the server for it to determine what to do next? How is it doing that? Via events it seems, but those events call JS functions I presume...

Also, if I change content is it doing incremental updates with DOM mutations or is it conceptually closer to a get or post? And so on.

Can you provide some conceptual explanations in the docs for some of the ways you can use clog (especially as it relates to other web applications)? Descriptions I've seen online have been calling this an isomorphic web framework, but that doesn't really sound right either. My impression of those is that pages can be rendered server-side, but that's only done as part of bootstrapping after which they run (almost) entirely on in-browser code.

I would love to give this a try for a project I'm working, but it's like an e-reader, so it's much closer to a standard webpage than a desktop application which means the first problem I'm faced with is persistence. If someone drives through a tunnel or they live on a farm with poor cell service and they lose their connection and need to start over, how would I go about solving that? How much of it can be handled by clog or other components and how much would I need to write myself?

[rabbibotton]

I really appreciate your taking the time to dig in to CLOG. I hope I can provide clarifications that best represent what CLOG can provide and were deficient to your needs to see if makes sense to meet them as a means for CLOG to reach a broader set of use cases.

Conceptual Purpose Provide UIs to applications ranging from small single user utilities to massive concurrent multiuser applications of which a website is subcategory.

Communication

1. CLOG's current transport is client/server, http for initial bootstrap and WebSockets for continued communication.
2. The initial bootstrap at a minimum is to establish the WebSocket connection is a http page and a javascript file (in truth can be combined).
3. Once communication is established all messages between server and client are in JavaScript. (JS over WS). Changes take place as fragments of the DOM, JavaSript calls, etc. That is transparent to the user in most cases.

Initial stat of UI

There are three ways to establish the initial state of the UI.

1. The initial boot file - this is identical to traditional webservers/website solutions and every familiar technology offered is available, the boot.js file is just included on the served page from the same domain due to WebSocket restrictions.
2. To build the initial UI from scratch using CLOG's APIs
3. A hybrid approach as used by the CLOG Builder panels. The created panels are composite components that deliver their base UI in a bulk write of HTML/JS to the browser.

Performance considerations. Method 1 and 3 offer performance advantages to Ajax but similar performance to long polling. Method 2 is clearly slower but has advantages for a completely non-html non-js code based.

Post Initialization

CLOG offers an event based framework. In the next week or so, and will be part of CLOG 1.2, I will be releasing an additional "presentation" like system and intend on pursuing some other additional models of UI programing as well (I have been designing UI interfaces and systems or 20 years all of which are still in production and in some languages like Ada are the defacto standard for win32/64 and GNOGA that CLOG is based on. GNOGA is mainly used for embedded UIs but has been used for last 9 years for web applications and websites as well).

Even if ignoring the advantages of using CLOG's framework, CLOG offers an additional transport to long polling, ajax, and generic WebSockets, ie. JS over WS and a Lisp based framework to manipulate any element on the page, respond to events on page and execute any JavaScript.

Reliability

CLOG is more reliable than traditional client/server, rpc based, orbs, etc specifically because it is designed on top of web technology. However the robustness of any application is based on its design not just the transports being used. Currently CLOG will reestablish moderate interrupts to connectivity, on the same machine survive sleep cycles etc. Full disconnects need to be planned for in the design of your system. A current snapshot of the UI state can be take currently but the state server side is dependent on your design. I do have some ideas and tool I plan over the next year in development. but that are conveniences in the end every app must plan for failures.

Scalability

CLOG apps can easily be used completely stateless, making use of post/get in the same way as any web framework (even using Ajax), as stateful highly responsive push UIs (something you can not do without JS over WS) with or with out state, or as responsive highly state oriented systems (again something unique to the CLOG approach for web apps).

Bottom line, the scale widely ranges based on app design, I can use CLOG to just serve pages static pages, server side constructed pages, server controlled construction on the client side pages, a hybrid of all those, and can respond with get/post, events, etc. In all cases the scalability is generally identical to a similar system for other web based systems.

However things that make CLOG unique from websites are systems where concurrent users are interacting, large scale business applications, or any application needing real time information delivery the scaling will more closely follow traditional client server applications.

Your example

In your example I would likely design the UI in builder, maintain key elements of state of the server side (username -> page in document) and in case the connection is bad enough that the normal recovery system in CLOG can't handle either automatically retry refreshes of entire page or just the boot.js, if needed have a fall back system for extreme users that is not as interactive and uses get/post. My experience though has shown in 10years using GNOGA/CLOG to design it using the base tech first. The scan and robustness issues are usually trivial to overcome since in the end it is still riding on the same web tech as everything else just abstracted away with CLOG.

[solarmist]

This is great. Please add this as a section to your documentation.

I honestly don't see a lot of places where it's obviously deficient, it's more that it's so different from the way I approach these problems it isn't obvious to me how to apply clog to the kind of app I want to build.

I think filling in these kinds of bits and bobs will really help with people taking a look at it and getting ideas about how they can use it.

It'd be great to add some examples to the tutorials of these kinds of hybrid apps. Or maybe a small series of tutorials starting from something resembling a traditional website and doing a couple of transformations on it into something using doing native updating of the DOM. That way they could see the same thing they're familiar with re-imagined in a few different ways.

I know you're eager to show off all the cool things that your library can do. But I think for most people seeing this for the first time it's too much of a jump and without being able to ground/connect it to what they're already familiar with it people will be lost at best or they will dismiss it or forget about it at worst.

[rabbibotton]

Excellent ideas. I will try to express in code some of these ideas.

[rabbibotton]

I have added CONCEPT.md to the repo and link from the README.md. Thanks!