[hiccup] extensibility/compilation API

[loganpowell]

Hi @postspectacular, I'm sorry for the long pause in activity (perhaps it was a breath of fresh air for you 😉 )

I have given a lot more thought to your advice of thi.ng as a collection of tools rather than a framework, which I understand now. Forgive me as I'm what people refer to as a "cargo cult'er", but I see how the thi.ng collection contains a broad variety of useful tools that I can apply to many of my projects, even if it's not a framework.

That having been said, I think there is a great opportunity for some integration points here and I was wondering if you'd be open to creating an API into hiccup (for starters) that could be used to transpile hiccup into another template (e.g., React.createElement). I would love to code my react components using hiccup.

[postspectacular]

Hi @loganpowell - yeah, thanks for the break! :) :) But seriously, I'm grateful for all the enthusiasm, new ideas / food for thought etc. I'm principally open to things leading to better integration with other tools, but simply don't know enough about React to do this myself (nor am I an active React user, so also couldn't prioritize this). That said, am open for PRs for such things, but since I'm also increasingly aware of the growing size of the code base and the related maintenance tasks, I'd maybe opt for such a project to be developed outside umbrella, with you (or others) as owners/maintainers.

Also, could you please provide some more context what such a transpiler would entail? If I understand correctly, you'd want to use hiccup instead of JSX and the transpiler would be AOT (ahead of time) during the project's build phase, or are there're other use cases?

[loganpowell]

I would be happy to maintain a separate fork if I can figure it out. It might make sense as a babel plugin

[postspectacular]

I'm happy to help & advice, of course! To get the ball rolling, it would be great to first collect some links about React/Babel internals and JSX options. I also foresee potential fundamental issues about transforming stateful hiccup components (i.e. closures) and handling of iterators/generators (which are used ubiquitously, at least in my own hiccup projects), even in stateless components. E.g.

import { comp, iterator, map, partition } from "@thi.ng/transducers";
import { serialize } from "@thi.ng/hiccup";

const img = (src) => ["img", { src }];
const link = (href, body) => ["a", { href }, body];
const imgLink = ({ src, href }) => link(href || src, img(src));

const thumbs = (images, columns) => [
    "div.thumbs",
    iterator(
        comp(
            map(imgLink),
            partition(columns, true),
            map((row) => ["div", ...row])
        ),
        images
    )
];

serialize(
    thumbs(
        [
            { src: "a.jpg", href: "/users/a" },
            { src: "b.jpg" },
            { src: "c.jpg" },
            { src: "d.jpg" },
            { src: "e.jpg" }
        ],
        3
    )
);

Results in:

<div class="thumbs">
    <div>
        <a href="/users/a"><img src="a.jpg"/></a>
        <a href="b.jpg"><img src="b.jpg"/></a>
        <a href="c.jpg"><img src="c.jpg"/></a>
    </div>
    <div>
        <a href="d.jpg"><img src="d.jpg"/></a>
        <a href="e.jpg"><img src="e.jpg"/></a>
    </div>
</div>

Maybe there's an easy solution to this, but as I mentioned, I don't know enough about React to say how this could be approached...

[nkint]

Hi!

Just to say my two cents. I love hiccup api and I'm using React everyday (and I like it too) but honestly I don't think they can be interchangeable in an easy way. Just think about some "core" React feature like hook for local state management or the new Suspense/React.lazy mechanism.

More over as toxi said, the transducers can be hard to port.

Anyway there is a big space for explorations (some random ideas following).

From umbrella to react:

• use JSX for hiccup can be nice for people coming from React or plain HTML,
• use the atom package for the global state management and make some React HOC to trigger re-render when an atom changes (like mobx-react-lite is doing).

From react to umbrella:

• types for the dom attributes: I really enjoy when react with typescript warn me that a prop named foo is not a valid prop for a <div> (Property 'foo' does not exist on type 'DetailedHTMLProps<HTMLAttributes<HTMLDivElement>, HTMLDivElement>'.) or when I miss some mandatory props... this is a little verbose for now in hiccup and I see some space to let it be easier to write,
• better error reporting (see #86 )

Some more ideas can be found in this brilliant talk Don't build that app where they are using ijk or domz or htm module to skip the JSX pre-processing step and write more hiccup-like React code. See snippets in the video here or here or here.

@loganpowell Did you have any other ideas?

[postspectacular]

Thanks for this, @nkint!

Just a brief note about the typed attribs - this is only problem if using the lazily evaluated component form (which admittedly is kinda default I've been using, but then again, not really) instead of direct component function calls. Here the issue is that TypeScript doesn't allow you to properly type array elements (or at least all of the ways I could think of ended in failure). Apparently there'll be some improvements in TS3.7, but haven't tried that yet. To illustrate better:

// arbitrary def for user context object
// see https://github.com/thi-ng/umbrella/tree/master/packages/hdom#user-context
interface MyHdomContext {
  theme: Theme;
  ...
}

// a component
const foo = (ctx: MyHdomContext, id: number, blah: string) =>
  ["div", ctx.theme.foo, `User: ${id} (${blah})`];

Call method A:

start(() => [foo, 42, "hdom"], { ctx: { theme }});

Pros:

• automatic context injection
• uniform syntax (also for component classes/objects with ILifeCycle impl)
• components are only data
• non-eager realization/execution (only during tree processing)

Cons:

• no type checking

Call method B:

start((ctx: MyHdomContext) => foo(ctx, 42, "hdom"), { ctx: { theme }});

Pros:

• fully type checked
• component functions don't always need extra initial ctx arg, can decide on-demand

Cons:

• generally more verbose

There's nothing in hdom/hiccup, which stops you from using both methods and I do that occasionally, even though I've never had real issues with the untyped method (A). VSCode is good enough to show me expected arguments when I hover over a child components name and my data is usually already mostly in the right format when it comes to building/updating UI...

Ps. I think the main aspect of this integration request here are really about hdom, rather than hiccup, since the latter is just a one-off serialization tool (largely meant for serverside or offline html/svg generation), without consideration for state change and/or time (axis) concerns...

[loganpowell]

I've looked deeper into the issue, considering what has been mentioned here. I suppose this is why I believe that the hiccup would need an API for creating such bridges rather than it be a simple syntax expansion/transpilation of hiccup to x (via Babel). What I was hoping for was an API that would allow for some kind of "hook" (not necessarily for React hooks, etc. forgive the play on words) into the hiccup form that would allow for the user (e.g., of React) to mix in the host frameworks bus (e.g., React hooks).

For example, from hdom:

["button", { style: {... }, onclick: () => bus.dispatch(["inc-counter"]) }, `clicks: ${state.value.clicks}`]

React:

const useCount = () => {
  const contextValue = useContext(CountContext);
  return contextValue;
};
const Example01 = () => {
  const [count, dispatch] = useCount()
  return ['button', {style: {...},  
    onclick= () => dispatch('increment')}, // <-injection point
    `clicks: ${count}`] // <-injection point
}

something like that

[loganpowell]

It looks ugly tho...

[postspectacular]

As for re-rendering when an atom's state has changed. This is easily done by attaching a watch to the atom and do some synchronization with RAF to avoid multiple re-draws if there're multiple state changes within the time span of a single frame:

import { IWatch } from "@thi.ng/api";
import {
    DEFAULT_IMPL,
    HDOMOpts,
    HDOMImplementation,
    resolveRoot
} from "@thi.ng/hdom";
import { derefContext } from "@thi.ng/hiccup";

// almost identical  to hdom's start(),
// but attaches as watch to given atom/cursor/history...
const updateUIWatch = (
    state: IWatch<any>,
    tree: any,
    opts: Partial<HDOMOpts> = {},
    impl: HDOMImplementation<any> = DEFAULT_IMPL
) => {
    let isChanged = true;
    let isActive = true;
    let prev: any = [];
    const _opts = { root: "app", ...opts };
    const root = resolveRoot(_opts.root, impl);
    const update = () => {
        if (isChanged) {
            isChanged = false;
            _opts.ctx = derefContext(opts.ctx, _opts.autoDerefKeys);
            const curr = impl.normalizeTree(_opts, tree);
            if (curr != null) {
                if (_opts.hydrate) {
                    impl.hydrateTree(_opts, root, curr);
                    _opts.hydrate = false;
                } else {
                    impl.diffTree(_opts, root, prev, curr);
                }
                prev = curr;
            }
        }
        isActive && requestAnimationFrame(update);
    };
    state.addWatch("hdom", () => { isChanged = true; });
    // initial UI
    requestAnimationFrame(update);
    // cancellation function
    return () => {
        isActive = false;
        state.removeWatch("hdom");
    };
};

const state = new Atom({});
updateUIWatch(state, app, { ctx: ... });

[postspectacular]

Can one of you please post some good links about this React hooks thing and/or explain what the big deal is about this? Thx :)

[loganpowell]

https://reactjs.org/docs/hooks-overview.html

however, when compared to the elegance of hdom, you might throw up...

The idea is to be able to "hook" into the React lifecycle via "functional" components (vs Class components). It's basically a way to plug into React's event bus that tracks when the dom should be re-rendered. There are a number of built-in hooks.

It's a kludge, kinda. They have legacy to support, so you have to jump through hoops ("hooks") to make the functional style work with React's lifecycle.

[nkint]

Ahhaha yes sorry 😅 It is a new way to handle local state to permit to write only function components (actually with not-pure function even if they claim to be more functional with hook) instead of class component and grouping logics that before were spread around lifecycle method in a single function.

https://reactjs.org/docs/hooks-intro.html https://reactjs.org/docs/hooks-overview.html

To better understand, a naive implementation of useState from Kyle Simpson.

Some good links: https://medium.com/@dan_abramov/making-sense-of-react-hooks-fdbde8803889 https://overreacted.io/how-are-function-components-different-from-classes/ https://overreacted.io/a-complete-guide-to-useeffect/ https://overreacted.io/writing-resilient-components/ https://overreacted.io/how-does-setstate-know-what-to-do/

Hooks have some special rules (like: "call in the same order" - react team also made a custom eslint rule to guarantee it) https://overreacted.io/why-do-hooks-rely-on-call-order/

[postspectacular]

Thank you both! Looks like I will do some reading over the coming days... :)

[nkint]

I also mentioned React.Suspense and IMHO it worth to read also this:

https://overreacted.io/algebraic-effects-for-the-rest-of-us/

In the end sometimes to simulate an async render they actually "throw a Promise", that is a weird concept, but this article worth the reading!

[loganpowell]

The primary reason I'm thinking about the React integration is it's Native target. You could just skip React entirely and just look at NativeScript. Having the native target is probably one of the biggest selling points of React.

I'm actually leaning towards Svelte 3 in this case because of this

Perhaps compilers will replace VDOMs in the near future. One benefit of compiler vs VDOM is that you can actually see what the compiler is doing to the code, whereas with VDOM it's opaque and more 'difficult to reason about'

I'm sorry, I realize that this is a derailment 😢

[earthlyreason]

Just to add to the point about type checking, recursive type aliases (coming in TypeScript 3.7) will allow for somewhat stronger typing of hiccup templates than is currently possible (or at least possible idiomatically), but they won't support anything that wouldn't already be possible through interfaces (if hiccup templates weren't array-based). In particular, checking component arguments against their signatures can't currently be done through data structures alone.

In other words, although you can write an interface like

interface Binding<T, A extends any[]> {
  fn: (...args: A) => T;
  args: A;
}

there's currently no way to write a type that implicitly enforces the relationship between fn and args in arbitrary locations, e.g.

interface Template {
  children: Binding<any, any[]>[]; // Forced to use `any` since this is not an inference site
}
// checks even though generics could be inferred if `children` were an inference site
const foo: Template = { children: [{ fn: Math.sin, args: ["bar"] }] };

The same is true for the more terse, Array-based expressions.

The workaround is to use a wrapper function that provides a call site solely for the sake of inference.

Theoretically (it seems to me), this sort of thing should be possible, and maybe Anders already has something in mind.

Anyway, apologies for the discursion. I'm in the slow process of abandoning React altogether, and this has been one of the blockers. We have a ton of code that benefits materially from prop checking. I am working on a transition that will preserve that, but again, it won't be a React-adapter, and the pain point is more a limitation of TypeScript than of hdom.

[loganpowell]

the fact that hiccup is 'just data' is it's primary selling point IMHO. This makes it very different from an interface, so this might just be trying to fit a round peg in a square hole...

I could imagine sending hiccup over a wire as JSON. Very little 'out-of-band'

[earthlyreason]

Absolutely, moving from opaque code to data is a major win. (And my ultimate goal is for all of this to be dynamic anyway, so static typechecking won't matter.) And note that the helper function in question just returns the argument array as-is, so the result is still transparent (unlike React elements).

If you look at the motivating example for recursive type aliases, it's clear that others are doing this as well:

https://github.com/microsoft/TypeScript/pull/33050

It's just not possible (yet) to capture relationships implied by function expressions. Incidentally, being able to do so would have implications well beyond hdom, as you could essentially write a form of typed s-expressions. (I am currently using the same technique to describe streams and transducers.)

So the balance for me is getting the benefit of data-based templates while preserving the ubiquitous static checks that have saved our skin countless time. Using a helper function looks like the best option for now, and it may not be necessary forever.

[loganpowell]

I respect your Pragmatism. Idealistically, values could replace semantics, but I realize this is very idealistic. If it's any consolation, reading these comments has already helped me frame the problem through a clearer lens.

[loganpowell]

Forgive me for beating a dead horse here, but humor me:

Would it be feasible to use the App entry point e.g., ReactDOM.render($Hiccup(App), node), or start($Hiccup(App), node) or svelteNative($Hiccup(App)) as a signal for a compiler to translate the hiccup imports to the target implementation syntax? Instead of trying to parse the entire projects AST for a specific syntax? This might require an AST API which would take the hdom functions and 'put them in the right place' for the target framework, allowing the hiccup Arrays not to have to be sniffed out from the 'noise'.

Maybe the holy grail of "Write Once ..." or I could just be an idiot 😅

Forgive me if this sounds completely idiotic.

[loganpowell]

I'm going to close this. I'm way out of my league here 🤣

[loganpowell]

Sorry I spazzed out. I'm looking at this:

https://github.com/thi-ng/umbrella/blob/master/packages/hdom/src/api.ts#

Should be useful for this purpose. I realize that you guys probably think - by now - I'm an absolute tool and I wouldn't blame you for it after the schizophrenic orgy of topics I've brought up in this issue. I am having a hard time finding the right words to express the opportunity I see for the project.

Thi.ng is so different from everything else out there. While it seems the status quo is template-all-the-things, the thi.ng philosophy is so simple, it evades easy analogy. While I can say "just use data" it's hard to unpack how much better that is than all these little micro-languages that are springing up.

Hiccup is the future. I'm sure of that. It's the most concise way to represent presentation as data. We need to prove that to the cargo-cults. They are also the future, for better or worse.

Just to unpack some context of my crazy thoughts here:

There's a new kid on the block, Svelte, that - instead of using a VDOM - is a compiler that uses direct DOM manipulation (safely - via the compiler) to improve performance. While the current benchmarks look promising, Svelte is killing it in the benchmark game. The compiler idea itself is enough to give people the 'shiny object' syndrome. But the performance is what's going to make it a business case.

I know @postspectacular has had some of the same experiences with Clojure as I've had (performance issues). I feel safe in saying that we here are Pragmatists and know that mutability (as long as it's not shared) is not evil. We want to make things that work and as fast as possible. More context: If we could combine speed with versatility (custom targets), it would be very exciting indeed.

[loganpowell]

you look at the motivating example for recursive type aliases, it's clear that others are doing this as well:

microsoft/TypeScript#33050

It's just not possible (yet) to capture relationships implied by function expressions. Incidentally, being able to do so would have implications well beyond hdom, as you could essentially write a form of typed s-expressions. (I am currently using the same technique to describe streams and transducers.)

Would it be prudent to jump on the bandwagon before it leaves the station?

I'm actually a typescript skeptic, but this PR is making me reconsider.

[loganpowell]

I have read-only experience with TypeScript, but it seems to me that - with this PR - by hooking into the TypeScript compiler API, hiccup/hdom could be transformed into arbitrary targets, or am I wrong?

[earthlyreason]

@loganpowell, this is an area near to my heart, so for my part the discussion is quite welcome. I will add some thoughts at the next opportunity (prepping for a demo at the moment), but just wanted to say that I'd always rather engage in some healthy problem finding than continue to try solving the wrong problem. This project has been a massive boost in that area, so I'd add my thanks to @postspectacular as usual.

[loganpowell]

@gavinpc-mindgrub thank you for your patience. I look forward to your thoughts!

DIgging into this a bit more, looking at the TypeScript Compiler API, it seems that API is rather undocumented/unstable ATM, but it is very exciting to say the least.

I did find these:

https://github.com/woutervh-/typescript-is https://github.com/cevek/ttypescript

[postspectacular]

For time reasons, this is not a full answer, just some interjections:

1. IMHO hdom doesn't qualify/classify as a VDOM, since it's completely stateless and the normalized version of the last user supplied tree is the only piece of data kept around (purely for API/UX convenience) until the next update happens. That's it. Normalization here merely means that all elements are in a uniform structure afterwards (i.e. ["tag", { attribs }, ...children]). There's no other internal representation of the given hiccup structures (like VDOM nodes, DOM references etc). Nada. The diffing is directly applied to these 2 arrays (prev & current tree) and the target data structure (e.g. browser DOM) is then patched via the current element's configured HDOMImplementation. However, hdom itself knows nothing about that target, nor does it want/need to. It's dealt with in an entirely opaque manner. Also the default HDOMImplementation itself is stateless and simply acts as abstraction layer for diffTree() to modify the target. As mentioned in the readme, currently there's only one other HDOMImplementation, the one used by hdom-canvas, which translates hiccup to canvas API draw calls...
2. That linked PR about recursive types in TS 3.7 is what I mentioned earlier on and am very excited about it. But I'm not quite following what purpose a TS compiler plugin would have in the context of this discussion? It's possible I'm missing something here... (if so, sorry in advance! :)
3. I've toyed around with Svelte a few moons ago, but the thing I'm still needing to wrap my head around is how/if it works with dynamically constructed reactive dataflows (rather than statically declared reactions in the AOT compiled source). The examples in the playground are not really touching this aspect... By keeping things like reactivity / state handling separate, hdom users are free to choose their preferred state management...
4. Maybe somewhat unrelated here, but I think hdom can never directly compete with all these other frameworks, unless it starts providing some sort of default state/event handling, nor does it favor component-local state. So a lot of the issue and solutions existing for React don't directly apply here. Keeping these things out was actually one of the main points (apart from loving hiccup syntax) for me, but I also understand that it's a real downer / factor of confusion for newcomers... Of course we could include some sort of wrapper around atoms and/or interceptors as default, but personally, I don't want to be married to them. I think state/event handling are entirely separate problems to UI generation/updating... I also know some of you are comparing each hdom feature to what you know from React, but the fact is that hdom operates somewhat quite differently and because of this also requires some unlearning and I don't mean this in any negative sense...

With all that said, I can definitely imagine a form of hiccup-to-JS compiler and will do some basic transformation experiments as soon as time & other priorities allow for it...

Keep these ideas flowing. This all is *good stuff! Thank you!

[loganpowell]

Hi @postspectacular!

1. I have mistaken VDOM for a diffing algorithm. I didn't realize it needed to be stateful in order to be considered such, apologies.
2. I figured, that - instead of having to track the entire DOM representation (hdom tree) on every change - a compiler transform could allow updates/mutations to happen in a tree-local manner. For example...
3. If you toggle the Show me button in this Svelte tutorial, you'll see that they use awfully ugly syntax to track local changes. I figured a simple deref of thi.ng's Atom or an event via bus could be injected as a function during compile and directly trigger dom mutation locally. Forgive me if this sounds ridiculous.
4. I respect your perspective here. Even though it hurts a bit to admit it, you'll probably get a better crowd if you don't accommodate the lowest common denominator (I fall into this camp atm with JS) by doing all the thinking for the user. That having been said, I am finding it difficult plugging the libraries together in some cases.

[earthlyreason]

Thanks @postspectacular for fleshing out more of the thinking behind hdom. I for one was indeed thinking of lifecycle components as offering something like React component state, and as a result of this misunderstanding have been unable to use them effectively.

But to back up a bit.

I have come to understand the philosophy behind thi.ng as what its name suggests: reification.

At the time when I encountered this project, I believed that the core scourge of the computing medium was invisibility-by-default, and I was dedicating all of my efforts towards a way of computing that made discoverability, visibility, and other properties that we expect of physical things to be inherent in the basic userland model. From these, plus the ability to take things apart, we could extend the model to include inspectability, portability, recomposability, and others. If you step back and look at what we do every day as software engineers, not having those traits is insane. Until we have them, everything else is a distraction.

I should add that for me, this is not about "software development." This is about empowering people. The future of user empowerment must lie in dynamic, userland solutions. The move must always be away from code and into information. (So I skipped over e.g. Svelte.)

Last December, my research in this area led me two things which have brought these efforts into focus. The first was RDF, and the second was thi.ng. (I found rstream-query when looking for a triple store/query engine, then quickly saw that the project was much larger in scope.) Studying RDF, it became clear that the solution to most of these problems must lie in a runtime knowledge base. RDF gives us a uniform data model for talking about any domain. A single knowledge base can talk not only about domains of importance to our applications but also act as a repository of the all the mechanisms that the system has created and how they relate to one another. Working with thi.ng, meanwhile, I found that almost everything it provided was already in line with the core idea: that every "thing" we talk about when we talk about systems must be reified somewhere.

Ten months into this, one thing has become glaringly obvious: we need a first-class way of talking about "processes and systems." (See Alan Kay.) Most platforms don't have an application-friendly process model. Erlang has its actors, Go has its threads. But even these are based on code rather than dynamic, userland descriptions. In the web, we have no first-class notion of processes whatsoever (where process is, at minimum, a composable way of organizing a sequence of invocations).

In my view, this subsumes the subject of web views. The problem with our so-called components is not with describing a DOM fragment. There are many ways to do this. The real problem is that we have no common abstraction for describing processes and systems. All non-trivial models are processes, and any "views" of those models are themselves processes. It stands to reason that we should find "views" a difficult subject if we don't have a basis for describing processes (and systems thereof) in the first place.

Again, thi.ng gives us many tools for exploring better ideas in this space. And it has much more of a dynamic, userland focus than anything I've seen. But I am still discontent. The solution I am working on basically inverts the current model by putting processes on top. In some ways, it's like "hdom for processes," where components are essentially expressions that describe an intended process (in practice, subscriptions and transforms) and the mechanism diffs and keeps the dataflow up to date. With this—and a different approach to fragment composition—I hope the view problem will appear in a better light.

I hope this manifesto is of some relevance here. I will share my prototypes by year's end.

In the meantime, I appreciate this project and have enjoyed following it. After several decades of application development, there is one quality that I equate with mature thinking: discontent with the status quo. In that respect, I consider this an "elite" group :)

[loganpowell]

That was inspirational. I agree with you overall. While reading, I couldn't help but to be reminded of how AWS deploys it's SDKs. Basically, they have a bunch of JSON config data that are used to configure language SDKs.

I may sound cranky, but don't get me wrong. I'm only here because I believe in the project and author. I'm just playing devils' (lazy-man's) advocate when I mention cargo-cults. Really, I'm just trying to scratch my own itch and these thi.ngs get so close to those 'hard to reach places' for me.

To your point about first-class 'processes and system', that's what really draws me to thi.ng. I can see:

hdom

• passing presentations over wires,
• describing routes as an hdom tree,
• optics/lenses into an hdom tree with simple property accessor patterns,
• adorning state with hdom on the fly,
• arbitrary targets (e.g., native, static, APIs), etc..

transducers

• one set of functions that work with any event source (pull, push, broadcast, csp ...) instead of cottage functions for each implementation (streams, events, data, etc.) huge
• efficient processing with deterministic termination
• I can imagine transducers as lambda functions (data -> data)

It's just data after all. So much potential at the presentation and process level. It's a beautiful abstraction. I mean, not to be meta, but you can transduce an hdom tree 😄

I very much look forward to seeing your work later this year!