Building interactivity into the kernel

[hannobraun]

Introduction: interactivity

I've recently realized that the current design of Fornjot's CAD kernel leaves a lot of potential unrealized. As of this writing, the kernel is basically a batch processing system. You put in a list of geometric operations ("here's a sketch, extrude it along this path, then transform the resulting shape in this way"), and on the other side comes out a triangle mesh (in the future, also other formats like STEP).

This is problematic because you are just presented with an end result. You only have limited insight into how the kernel came up with this result, and how you can change the inputs to change that result. It would be better to be able to interact with the running kernel while it's doing its job, so you can better understand what it's doing.

If this isn't self-evident to you, I highly recommend watching Inventing on Principle by Bret Victor: https://www.youtube.com/watch?v=PUv66718DII

I know it's a lot to ask to watch an hour-long talk just to understand a GitHub issue, but I promise, if you're interested in programming at all, this is worth your time. Also, he gets to the part that's relevant to this issue pretty quickly, and you don't have to watch the whole thing to understand the potential.

I also highly recommend Stop Writing Dead Programs by Jack Rusher, which was the recent inspiration for my change in thinking here: https://www.youtube.com/watch?v=8Ab3ArE8W3s

What I want to change

I've always envisioned Fornjot as an interactive system, but I didn't think of the kernel itself as interactive. In my mind, the user would edit their code, the Fornjot app would call the kernel with the updated code, and the result on the screen would be updated. I always had this notion that, eventually, the kernel would need to carry over state between those invocations, but just to cache intermediate results for performance reasons.

What I want to achieve, is to make the Fornjot kernel completely interactive. Turn it into a system that you can feed input to bit by bit and see intermediate results at every step. Even replay events to recreate an old state, or roll back events to immediately see what came before.

How will this affect the end user? I honestly don't know. I'm still in the beginning stages of figuring out where all of this will lead. But I wouldn't be putting so much thought into this right now, if I didn't see a benefit.

Immediate benefit: debugging

Developing a CAD kernel is hard! I spend a lot of time trying to understand how a million little steps have been leading to an undesirable outcome, based on debug output that is either too verbose or lacking critical information. That's one reason that watching Stop Writing Dead Programs hit me as hard as it did. Because not only did I know, in principle, that things could be better (I saw Inventing on Principle many years ago and have been thinking about the topic ever since), I'm also suffering from them not being better.

Fornjot is a big project and I will spend much more time working on it. Anything that makes this work more efficient, will pay off in many ways. Things can get done faster, I will be happier doing them, and not least, making things easier on developers can help attract more of them.

Plans

While a lot of details still remain to be explored, I believe that the kernel should become a long-running service that can be inspected at every level. I believe the way to do this, is to separate it into a number of isolated services, each of which wraps a piece of state and uses event sourcing as the only means of modifying that state.

The events produced by each service will be available outside of the kernel, and can there be used as the primary means of inspecting the kernel state. For example, the Fornjot app could be put into a debug mode where it displays a list of events produced by each service. Selecting any event from a list could result in a visual representation of the state of that service, as of this event.

What these visual representations look like, would depend on the state being shown, and there could be multiple representations per type of state. For example, when inspecting the objects that make up geometric shapes, these objects could be shown as a graph (each object being a node, references between them being edges) or a 3D representation of the shape, similar to how shapes are already displayed. Each of those representations could be useful, depending on what specifically you're looking for.

Prioritization

Implementing this could take a very long time, and I'm acutely aware that it's just one of many competing priorities. The most important thing is to make Fornjot useful to people (because if it isn't, there's not much point in its existence), and the most important feature that would move us into that direction are boolean operations (#42, #43, #44). Work on those is currently blocked, I'm already two layers deep into a pit of distraction, and I don't need to go down a third.

However, I do believe this is important, and my work would benefit immediately from better debugging tools being available. To bridge that gap, I'd like to start introducing this change piecemeal, so it doesn't block other things, and we can realize the benefits as we go. For this purpose, I've been working on a prototype, to figure out a suitable design. Here's my current work-in-progress: https://github.com/hannobraun/Fornjot/pull/1387

Next steps

Next, I intend to finish my prototype and deliver a vertical slice of this concept. This will show us, if the whole idea has any legs at all, and what needs to be adapted. Later on, I'd like to keep working on this on the side, probably in response to specific problems I'm having. This should hopefully guarantee that the kernel architecture moves into the right direction over time, without blocking progress in other areas.

Specifically, I'm thinking about the following:

1. Initial prototype: Create a service that wraps the Objects struct. This should be relatively straight-forward, while already providing a lot of insight into how the kernel evolves its state.
2. Create a service that handles validation. Right now this is built into Objects, and a validation failure results in an error that stops everything. Having validation happen in a separate service instead would make a lot of code simpler (because object insertion becomes infallible again), while providing more insight into which validation failures are produced in a given situation (beyond a print-out of just the first one).
3. Create a service that handles partial object construction. This is closely related to #1249, and I'm honestly not sure how exactly to approach this. I can't really design this service without solving many of the problems that the partial object API has, but I would benefit from the insight that this service can provide to solve these problems. Maybe I can come up with a good approach. We'll see.

Not sure where to go beyond that, but the work up until this point will generate a lot of new insights.

[hannobraun]

I've opened a draft PR with my current design for the service infrastructure: https://github.com/hannobraun/Fornjot/pull/1387

[MartinKavik]

Just a list of possibly interesting links related to interactivity and programming languages in case you would like to introduce some advanced UI controls into Fornjot or want to create your own language or integrate existing languages.

The list

- Enso - https://enso.org/ - https://enso.org/language - "Hybrid visual and textual functional programming." - Data processing where you can define the flow through the nodes or text. - Roc lang - https://www.roc-lang.org/ - https://www.youtube.com/watch?v=6qzWm_eoUXM&t=3508s - https://www.youtube.com/watch?v=9f12S001TGQ - Typed, functional lang. Runnable with errors, compilable to Wasm, don't need type annotations, should support hot-reloading, they are writing their own interactive editor with first-class Roc supports, supports abilities, has REPL. It's WIP. - It could be a interesting alternative language + editor for Fornjot models. - Rhai lang - https://rhai.rs/ - Scripting lang embeddable to Rust or to Wasm. - Hot-reload + hot-patching, preserving state between invocations, API for AST manipulation, sand-boxing.. - Could be interesting when we want to implement interactive code editor with real-time model changes even without our custom plugin system based on Rust crates or Wasm. - Dark lang - https://darklang.com/ - You are basically writing your backend logic, database definitions and crons in a browser tab in a boxes with code where you can interactively see values of the defined variables. - `futures-signals` - A Rust library for handling changing states and their relations in your app. It powers MoonZoon where it replaces things like VirtualDOM and state managers. - https://crates.io/crates/futures-signals - https://docs.rs/futures-signals/latest/futures_signals/tutorial/ - I can imagine it can manage for example model parameters - the user can freely change them and the app will automatically recompute dependent variables and then rerender the screen accordingly or save the data. - Flux - https://liquid-rust.github.io/2022/11/14/introducing-flux/ - I assume defining 3D models in Rust needs API with a lot of number operations, perhaps refinements could make such API stricter/safer. - Lunatic - Rust backend runtime based on WebAssembly, inspired by Erlang. - https://lunatic.solutions/blog/rust-without-the-async-hard-part/ - It also allows to write Rust code without `async`. - Node-based UIs - https://github.com/wbkd/awesome-node-based-uis - https://github.com/samuelmtimbo/unit - Unison lang - https://www.unison-lang.org/ - It hashes code and stores it in a database. It prevents dependency conflicts, allows hot-load, supports durable variable storage, no need to recompile projects, etc. - Pony lang - https://www.ponylang.io/ - No locks, no atomics, based on actors. Supports capabilities/abilities. - Kindelia - https://kindelia.org/ - Super fast virtual machine for functional languages and a language based on it. - Effects, Capabilities, and Boxes - Whitepapers + interactive examples about effects. - https://se.informatik.uni-tuebingen.de/publications/brachthaeuser22effects/ - Effects can replace exceptions, const/async, direct IO calls and other stuff with one concept. - Reachability Types - Something like Rust ownership on steroids. - https://www.youtube.com/watch?v=T_mDt-qnmMk - https://github.com/TiarkRompf/reachability - Magmide - https://github.com/magmide/magmide - "A dependently-typed proof language intended to make provably correct bare metal code possible for working software engineers."

[hannobraun]

Thank you for that list, @MartinKavik! Lots of interesting stuff on there, most of which I didn't know about!

[hannobraun]

I've implemented a first go at the design that the in-kernel interactivity can be based on, mostly in #1392 and #1404. So far this isn't realized yet, but at least some of the building blocks should be there.

In the interest of not taking too much time away from other critical areas, I will focus on something else now, but will keep working on this on the side, as needed. There are two obvious areas of improvement from here:

1. Make use of the new service-based architecture. So far, the produced events are just ignored, as there are no tools that can show any intermediate state or other valuable information.
2. Expand the service-based architecture. The obvious next step (as noted in the issue description) would be to create a partial object construction service, but I'm still thinking about how exactly this could be done.

[hannobraun]

I came across Makepad today: https://github.com/makepad/makepad

It's a UI framework that supports a feature they call "live design". Here's a demo that I find quite impressive: https://makepad.dev/

If you make changes to the code within that live_body! macro call, you see instant results in the "WorldView" thingy on the lower-left. This combination of Rust code with an internal DSL is certainly interesting, and something I hadn't considered for Fornjot so far.

[hannobraun]

I'm starting to open more specific follow-up issues for this planning issue. Here's the first one: https://github.com/hannobraun/Fornjot/issues/1613

[hannobraun]

I've opened another more specific follow-up issue: https://github.com/hannobraun/Fornjot/issues/1627

[MartinKavik]

It looks like Dioxus has rsx! that is similar to Makepad's live_body!. See "Hot Reloading" section on https://dioxuslabs.com/blog/release-030/

Perhaps a bit OT: I've just read about Ambient (https://www.ambient.run/post/introducing-ambient) - maybe there is a chance to leverage those live macros to add some collaboration features (think of Figma/Google Docs for 3D with automatic code sync). Just a wild idea :slightly_smiling_face:

[hannobraun]

It looks like Dioxus has rsx! that is similar to Makepad's live_body!. See "Hot Reloading" section on https://dioxuslabs.com/blog/release-030/

Nice find!

Perhaps a bit OT: I've just read about Ambient (https://www.ambient.run/post/introducing-ambient) - maybe there is a chance to leverage those live macros to add some collaboration features (think of Figma/Google Docs for 3D with automatic code sync). Just a wild idea :slightly_smiling_face:

I honestly think that this is the kind of thing that any programming environment should offer. We're not there yet (as an industry in general, and especially not with Fornjot :smile:), but I kind of expect/hope that this kind of direct collaboration will just be expected at some point.

Whether I want to make it work in Rust or whether we'll use some other language for the higher-level stuff by then, I don't know.

[hannobraun]

I've decided to close this issue. Since Fornjot is just a kernel now, no longer an app, there's limited infrastructure that could provide the kind of interactivity I'm imagining. I hope there will be opportunities in the future to still do this, as it would make developing Fornjot much easier, but at this point, I don't have a clear vision.

If I can come up something in the future, I'll open more targeted and actionable issues.