Alternative runtimes

[mathisonian]

It would be nice to make the React dependency more modular. All of the React specific code lies in the idyll-components and idyll-document modules.

The components are implemented in react and would need to be re-implemented using whatever other framework one wanted to plug in (unless there is a meta component specification that exists and works across frameworks?)

The runtime (aka idyll-document) needs to read in the AST and instantiate the application. We'd need to to some work documenting all the behavior to make it feasible to re-implement with other frameworks, but I don't think it would be too bad.

/cc @eyeseast

[eyeseast]

Finally have a minute to think about this. I'd really like to figure out a way to use Svelte with Idyll. Here are a couple starting points:

Documenting the AST is probably the first thing that needs to happen. Maybe this is part of #358.

idyll-ast is probably another useful piece of the puzzle.

I'm not sure if makes sense to compile an Idyll file at runtime or compile time. In other words, is it better to port idyll-document into Svelte, with the same API, or to do something clever with svelte.preprocess and turn raw markup into a root component?

Porting idyll-components seems doable and necessary.

Assuming this whole idea works at all, is it worth adding alternate runtimes into the existing idyll CLI and toolchain, or is this an advanced use case that just assumes you need to write some javascript?

[mathisonian]

Thanks @eyeseast. Agreed that the first thing to do is clean up the AST documentation and potentially get #358 in as part of that. @manjhawar96 and I are meeting this afternoon to discuss the AST updates and documentation, which he'll be working on.

Here's a little more info to give you a better idea of how things work now.

The AST defines a tree structure, where each node of the tree conceptually looks like this:

{
   name: "p", // html tag or component name
   properties: [{
      name: "propertyName",
      type: "value",
      value: "Hello World"   
   }],
  children: [] // an array of other nodes
}

Note that this looks very similar to an HTML tree with a major difference being that

• A node can refer to a JavaScript component or variable definition, rather than just an HTML tag
• The properties can have different types.

The various types are:

1. Static values. e.g. [a href:"http://..."][/a]. The href is a string that never changes.
2. JavaScript expressions, e.g. x squared is [Display value:`x * x`]
3. Bound variables, [Range value:x min:0 max:10 /], this allows the component to update the bound variable.

It is the job of the runtime to convert these AST nodes into javascript, and make sure that they stay in sync with the current variable values. With React this means dynamically generating a react component tree at runtime that behaves as expected. We generate this same tree on the server and in the browser to achieve SSR.

I suspect you'd want to do the code generation outside of the browser and just let svelte do its thing to generate the JavaScript that actually gets loaded.

If we get this up-and-running I'd want to add an option to the CLI to specify which runtime to use, nearly all the CLI logic is framework agnostic, there'd just be a few places where we'd have to add some logic depending on whether react or svelte is being used.

[mathisonian]

I'll write out a step-by-step algorithm for the logic that the current runtime implements. Is there any suggested reading where I could learn more about how Svelte (and preprocess) work under the hood?

/cc @eyeseast @Rich-Harris

[eyeseast]

Super helpful. If I'm understanding this right, you could almost write a runtime in vanilla JS and just treat each component as a black box. Coordinating bound variables is probably the hardest thing, and the biggest gain from having a framework.

I don't know if there's a good resource for Svelte under the hood. The docs are good: https://svelte.technology/guide. The big difference from the other frameworks is that it does most of the work at compile time, instead of run time. There isn't a way (that I know of) to make arbitrary elements outside of templates, but maybe you don't need to. Maybe it's just a helper function that turns an AST node into a DOM element (or renders the component).

I haven't actually used svelte.preprocess in a project yet. The idea is that you can write your components in something other than plain HTML/CSS/JS and then transpile what you have into a proper Svelte file. Maybe converting an Idyll AST into HTML just so Svelte can convert it into vanilla JS is too clever.

[mathisonian]

Yes, I think that's right @eyeseast - I could imagine extracting the runtime to vanilla JS, although we'd have to think through exactly what the API would need to look like in order to be consumed by React/Svelte/etc. I'm not sure if there are examples of existing APIs that offer something similar, but it essentially is just a global variable store that needs to be able to trigger component re-renders when a variable updates. Given that this is relatively straightforward to implement in all the modern frameworks I'm on the fence about whether its worth it or not to abstract it out.

I imagine if we can figure out a way to get svelte working off of the AST directly (rather than first generating HTML) there would be less headaches. We've been doing some work on making the AST more user friendly in #370, I think it would make sense to do a deep dive into how the Svelte render would function in practice once that lands.

[jarmitage]

Hi @mathisonian @eyeseast, any updates regarding this? Are you still eventually planning to make it possible to e.g. use Svelte with Idyll? Or has this already been happening elsewhere?

[mathisonian]

Hey @jarmitage - this never really took off. I think the best short term solution would be to create a version that allows users to write svelte components while keeping the React runtime.

This wouldn't be optimal in terms of page size, but I don't think would be too difficult to implement and would probably be immediately useful for folks who prefer svelte. We'd need to update our compilation pipeline to detect svelte components, and handle them appropriately in the JS bundle step https://github.com/idyll-lang/idyll/blob/master/packages/idyll-cli/src/pipeline/bundle-js.js.

[jarmitage]

Thanks for the reply @mathisonian, that does makes sense. I would definitely find the short term solution you've proposed useful.

[mathisonian]

Sounds good @jarmitage, thanks!

One thing that would really help since I'm not yet familiar with all of the ins and outs of svelte is what needs to be done in order to compile a svelte component from a .svelte file into JavaScript code which we can pass properties and attach to a DOM node.

[jarmitage]

Yeah unfortunately I'm no expert either, but there might be some answers in the Svelte Language Tools repo.

If I find anything else out I'll report back.

[mathisonian]

Okay thanks @jarmitage! Maybe @eyeseast would also know something about this.

[eyeseast]

Hi all. I fell off this a while ago. I still like the idea, but I haven't been able to dig into it. Svelte did a major rewrite shortly after this, which may or may not change how this would work. But I've been away from the idea for long enough that I'd really have to start from scratch to get anywhere. Sorry.

[mathisonian]

Hey @eyeseast no worries thanks for the heads up.

I'm reaching out to the svelte folks now to learn a little bit more about the architecture and will post here as I find out more.

[mathisonian]

Okay - small update.

I'm able to get svelte components compiling inside of idyll by using the sveltify plugin found here to transform them to standard JS at build time. However at this point it still requires that you have a React component that wraps it, for example, I've been using this one:

const React = require('react');
const SvelteComponent = require(`./my-svelte-component.svelte`);

class CustomComponent extends React.Component {

  componentDidMount() {
    const { idyll, hasError, ...props } = this.props;
    this._component = new SvelteComponent({
      target: this.ref,
      props: props
    })
  }

  componentWillReceiveProps(newProps) {
    const { idyll, hasError, ...props } = newProps;
    this._component.$set(props);
  }

  render() {
    return (
      <div ref={(ref) => { this.ref = ref }} />
    );
  }
}

module.exports = CustomComponent;

As a simple wrapper for svelte components. Note that this requires a small tweak to the current idyll build step, so it won't work if you try to compile this with the latest idyll code. We would need to incorporate the automatic generation of such a component for each svelte component, either in the idyll build step, or as a plugin, in order to make the developer experience reasonable.

[jarmitage]

Ok, nice work! I wonder if there will be edge cases with such a wrapper, does it seem fairly robust? Happy to test out some different scenarios if you can add this to a branch.

I’m not sure how you’re set up with idyll in terms of plugins and how that impacts maintainability, but maybe this could be encapsulated with a build flag, e.g —with-svelte?

[mathisonian]

I've found a couple edge cases but otherwise it seems robust. These are:

• Server side rendering isn't currently supported. I think there's probably a workaround where we can render everything except for the Svelte components ahead of time.
• Nesting isn't supported, and I'm not sure that it can be... I need to think about this one more. For example, normally with Idyll you can do something like

[MyCoolComponent]
   Some text here
[/MyCoolComponent]

and then have that component render out the children. However, those children are still in reactland. I'm not sure there's a way to have a tree that goes React ---> Svelte ---> React... maybe??

[mathisonian]

The nice thing is that everything else seems to work really well, including all the reactive variable binding.

[joshpoll]

Disclaimer: I'm not super familiar with the inner workings of either Idyll or Svelte, and just barely familiar with Svelte, so please let me know if this is off base!

I'm wondering if it's possible to rewrite Idyll as an embedded DSL in Svelte. That way Svelte could cleanly take care of all the reactive programming logic, while Idyll could be a nice markdown + default component layer on top. Another advantage of a Svelte backend over the current React-based backend is it avoids the updateProps dependency injection that makes plugging in some React components tricky. (updateProps also seems to mess with React components that use hooks.)

Another option besides an embedded DSL could be to compile Idyll files directly to Svelte programs. This would have the advantage of maintaining the current Idyll syntax. (I think this was suggested earlier in the thread.) It would require some logic to lower the variables in Idyll to variables in Svelte, but it might be the case that a "dumb" translation would work because Svelte could calculate the variable dependencies automatically.

[jarmitage]

@joshpoll I would be super interested in that!

[joshpoll]

Awesome @jarmitage! I've thought about this a little bit more, and I think the embedded DSL route is very promising. There is a markdown preprocessor for Svelte called MDsveX. That means Svelte + MDsveX could probably handle reactive values and mixed markdown-svelte syntax. (As an added bonus, these mixed components can be easily nested, which I'm not sure is possible with Idyll files.)

The biggest hurdle to this approach would be porting the Idyll component, layout, and theme libraries to Svelte. Since the themes and layouts are essentially CSS, porting them should be relatively easy. Porting components could be a little more difficult, but there are probably ways to leverage or customize existing Svelte components where possible. For example there's a Svelte scroller component.

@mathisonian @eyeseast do you have thoughts on the viability of this?

[jwilber]

Just checking in to see if there’s been any update on this/development of a svelte-with-idyll (can’t seem to find anything)?

[joshpoll]

Hey @jwilber. Matt and I discussed this further offline, but we're both busy with other things at the moment so I'm not sure we'll be able to work on this any time soon.

I looked through my email and found a snippet I wrote to Matt, which I've copied below. I made a very small proof of concept at the time, but set the repo to private. I just made it public so hopefully someone can pick this up if they'd like!

---

I've ported many of the simpler components and somewhat replicated the stacked scroller tutorial. You can find the repo here. To run it, clone the repo and run

npm install
npm run build
npm run start

App-Test.idyll contains some examples of other components. If you rename the file to App.idyll, you can run that one too.

---

Hope this helps!

[jwilber]

Great, thanks a lot Josh! I'll have a look at this