More modular architecture

[gossi]

Hey,

I was wondering if the architecture in here can be a bit more modular in order to support more use-cases. For instance, ec-addon-docs is running their own fork of this code and fixing errors on one project doesn't fix them on another.

On the other hand, I'd like to have the opportunity to write PlantUML in markdown (see ember-learn/ember-cli-addon-docs#399 but not only within ec-addon-docs).

I think both use-cases can be support without ec-addon-docs needing to do anything, but providing a good architecture from this plugin.

Given the nature of marked, there is a couple of things to hook into. E.g. this addon would use the regular lexer but on top would take the token stream and handle handlbars syntax with it (since this is special to ember/glimmer). It would also provide an interface, so other addons can provide their own transformations (on tokenstream, parser or renderer level). For example, I could write an addon, that handles PlantUML. Another addon can be to parse a toc. This information is collected and each consumer (app or addon) can hook into the parsing step (on cli level). For that it will receive an object with all meta information and returns the code it wants to. See my recent addition:

https://github.com/ryedeer/ember-cli-markdown-templates/blob/04535e356cd3daddd7f575ff5760176e12e5ec9c/lib/compile-markdown.js#L57-L59

Now, going back to the original idea, using PlantUML within ec-addon-docs. I would just install it and as a second package would install my PlantUML transformation addon, too. I can achieve my goal and ec-addon-docs is not required to do anything. PlantUML was just an example, yet there can be more for sure.

cc @samselikoff

[ryedeer]

Hello Thomas!

These plans are pretty ambitious for this addon that started just as a simple wrapper around a markdown library :) If you have ideas about this addon and the motivation to implement them, I can just add you as a collaborator to this repo so the process would be much easier. Please let me know if you're interested.

[samselikoff]

@gossi From a quick reading it seems like a fine approach to me!

As I've stepped back from working heavily on AddonDocs, one thing we might want to consider is to take a note from the Gatsby playbook and get a stronger general interface designed for getting data into AddonDocs. Gatsby has a concept of a "content mesh", and any plugin can add to that mesh with some pretty simple APIs. Then, on the frontend side, any React component can query that mesh using a static GraphQL query.

We don't necessarily have to use GraphQL, but it might be worth thinking about a more generalized AddonDocs "data repository" or content mesh, which all of our tools plug into: the different app versions, the guides content themselves (whether from Markdown or elsewhere), the API docs, the NPM version and other things from package.json, and so on. That would solve a lot of the open issues folks have who are looking to extend AddonDocs. Just throwing it out there as a thought.

If you want to work on this specific problem though I can definitely provide any feedback & help get it merged.

[gossi]

I think, the idea would be to do something like ember-cli-markdown, that only has the functionality of compile markdown to html and provide meta around (like a toc). Addons like ember-cli-markdown-templates can use that mentioned plugin to transform template.md to template.hbs, without having to worry about the compile step in general.

To customize markdown compiling any consumer of ember-cli-markdown can provide options for more compilation customization or proxy the method in total. I thought to use markdown-it instead of marked, from my research today, there is already more available that is ready to use and has a lot of plugins.

Will give it a try soon.

[gossi]

Update on this:

I switched from marked to markdown-it it has a better ecosystem and is easier extendable. I created markdown-it-ember which parses ember template syntax within markdown and persists it - turns out my regexes do a good job here (tested it with a private project that makes heavy use of that). Also vscode is using markdown-it and the preview support is just amazing, DX is impressive with this!

I'm unsure about my idea for ember-cli-markdown I still like it but I'm doubt on how to do it. Basically, all it needs is this (spin-off of compile-markdown.js):

// lib/markdown-compiler.js
'use strict';

const fm = require('front-matter');
const MarkdownIt = require('markdown-it');
const toc = require('markdown-toc');

class MarkdownCompiler {
  constructor(config) {
    this.config = config;

    const md = new MarkdownIt('default', config.options);

    const plugins = config.plugins || [];

    for (let plugin of plugins) {
      if (typeof plugin === 'string') {
        plugin = [plugin];
      }

      plugin[0] = require(plugin[0]);
      md.use(...plugin);
    }

    if (config.configure) {
      config.configure(md);
    }

    this.md = md;
  }

  compile(source) {
    const content = fm(source);

    let html = this.md.render(content.body);

    content.html = html;
    content.toc = toc(content.body).json;

    if (this.config.format) {
      html = this.config.format(content);
    }
    return html;
  }
}

module.exports = MarkdownCompiler;

With markdown-template-compiler.js slightly changed to handle this:

// lib/markdown-template-compiler.js
'use strict';

const stew = require('broccoli-stew');

const MarkdownCompiler = require('./markdown-compiler');

module.exports = class MarkdownTemplateCompiler {
  constructor(options) {
    this.name = 'markdown-template-compiler';
    this.options = options || {};
    this.compiler = new MarkdownCompiler(options);
  }

  toTree(tree) {
    const compiled = stew.map(tree, `**/*.md`, string =>
      this.compiler.compile(string, this.options)
    );

    return stew.rename(compiled, '.md', '.hbs');
  }
};

And by that it is configurable like this:

// ember-cli-build.js
{
  'ember-cli-markdown': {
    options: {
      linkify: true,
      html: true,
      typographer: true
    },
    plugins: [
      'markdown-it-abbr',
      'markdown-it-anchor',
      'markdown-it-deflist',
      'markdown-it-ember',
      'markdown-it-highlightjs',
      'markdown-it-ins',
      'markdown-it-mark',
      [
        'markdown-it-plantuml',
        {
          openMarker: '```plantuml\n@startuml',
          closeMarker: '@enduml\n```'
        }
      ],
      'markdown-it-sub',
      'markdown-it-sup'
    ],
    format(content) {
      // do something postprocessing with that:
      // content {
      //   attributes: frontmatter attributes,
      //   toc: structured table of contents,
      //   body: original markdown,
      //   html: html so far
      //}
      // return html;
    },
    configure(md) {
      // custom configuration for markdown-it
    }
  }
}

The MarkdownCompiler class compiles the markdown and understands the frontmatter, scans a toc and collects some information. plugins is a shorthand to configure the markdown-it extension you might wanna use and configure right there (see the markdown-it-plantuml example). For further configuration use the configure() method (e.g. this is where ec-addon-docs could mount their custom renderer wishes (or use an already ready markdown-it plugin and configure it respectively - if there is any). As a consumer, if you wish to use plugins, install them on your own and add them in your ember-cli-build.js giving this flexibility to consumers.

Basically, the MarkdownCompiler is pure javascript, so no need to make it an ember addon. Yet there is a chance this can be a convention amongst ember addons to use that. For example: ember-cli-markdown-templates will transpile .md to .hbs using the information in ember-cli-markdown key. Another addon ember-cli-markdown-resolver is similar to this addon (they share a little I guess) but both could benefit from a shared config (or build on top of each other?). The ember-guides are also build by parsing markdown (can't find the right repo atm).

I'm pinging @willviles and @mansona for this.

Maybe an idea (that one formed while I was writing this):

• Create a package just for the MarkdownCompiler class (should actually be `MarkdownItCompiler´ 😂)
• Create ember-cli-markdown which holds a shared instance of MarkdownItCompiler, ready to be used by various addons.

Suggestions please 🙏

[gossi]

Ok, one thing already happened: markdown-it-compiler 😂

[josemarluedke]

@gossi This proposal seems super interesting. Any other movements on this effort? RE: ec-addon-docs.

[gossi]

At Emberfest I had the chance to talk with @mansona about it and how it might find it's way into empress (or one of its product). He (or better precise robert) has an idea how to compile this at runtime. The idea here is, that with support from empress, you can have an API to load it as needed instead of compile it.

[mansona]

So one of the things that is different about my approach is that I'm looking to solve the problem of compiling Ember Templates at runtime instead of build time 🤔 The Empress product that @gossi is talking about is called Field Guide (as it's a beta product the readme isn't filled out but you can see it in action on the new ember-styleguide).

The solution that I was discussing with other Ember Core Team members was that we might be able to come up with an intermediate representation of the compiled handlebars so that we can support runtime execution without the need to ship the entire handlebars compiler in the bundle (adding at least 150KB in the process 🙈)

I'm happy to help in the effort to improve the overall support for Markdown in Ember but I think I will be coming back to the "mainstream" way of doing it after about 6 months of experimentation with my runtime efforts

[astronomersiva]

Here's what I have learnt from a parallel experiment that I have been running. That one uses remark instead of markdown-it and the only post-processing that needs to be done in the output is replacing < with <. It also supports plugins so it should be possible to use a list of plugins provided in the host app's ember-cli-build.js as well.

My experiments were more towards having both the template and the backing JS in the same file. While setupPreprocessorRegistry can be made use of to handle the template compilation part, I had to essentially write a hack with the treeForApp hook to extract and write the JS into a separate file. While this works, rebuilds do not happen for the backing JS and they are only triggered for the templates.