Support basic charts - Githubissues

damithc commented 3 years ago

Suggestions: Provide support (likely through a third-party plugin) to add basic charts (e.g., bar/line/pie ... charts) using text sources (or csv files), similar to how we support basic UML diagrams.

(Updating the description to include the TODO from the attempt in #2052

[x] Add optional mermaid plugin for alternative diagram & chart support

(Updating the description to include conclusions/TODOs from the discussions below, see https://github.com/MarkBind/markbind/issues/1663#issuecomment-1080122849)

[x] Include documentation/examples on how it is possible to utilize some of these charting libraries
[ ] Extract out a few common chart types with minimalistic options to allow for really simple use cases.
- E.g. a way to create a piechart by doing something like <pie-chart>data = [1,2,3]</pie-chart>.
- The user need not know what is the underlying charting library in use, and the user is limited in terms of the options provided.
[ ] Create a general abstraction to simplify the typical steps required

Elaboration on the last point:

import (help import the script according to the type defined)
define a location to place the chart (user define the data/option right here, or pass in from an external file via data attribute)
init with a data/option object (help pass the data/option and generate the required script to append to the page)

Example syntax:

<chart type="chartjs">
{
    type: 'bar',
    data: {
        labels: ['Red', 'Blue', 'Yellow', 'Green', 'Purple', 'Orange'],
        datasets: [{
            label: '# of Votes',
            data: [12, 19, 3, 5, 2, 3],
        }]
    },
    options: {
        scales: {
            y: {
                beginAtZero: true
            }
        }
    }
}
</chart>
<chart type="chartjs" data="/chart2.json"></chart>

tlylt commented 2 years ago

984
1079

I did a brief search and it seems like having integration with mermaid would be nice (Gitlab has native support for mermaid.js syntax). It supports pie chart, among other things.

Also, it seems like we were trying to integrate it (in #1079 @crphang will you be continuing on it?) If not I don't mind having a go at this if this feature is desirable

damithc commented 2 years ago

I did a brief search and it seems like having integration with mermaid would be nice (Gitlab has native support for mermaid.js syntax). It supports pie chart, among other things.

Thanks for looking into this @tlylt I don't mind supporting mermaid. My concern is that it has a large overlap with plantUML that we already support. The only extra thing I could spot was the pie chart. Still, we can support it if the cost is not too high.

A higher priority for me is to add support for more charts such as pie, line, bar, histogram etc.

Also, it seems like we were trying to integrate it (in #1079 @crphang will you be continuing on it?)

Unlikely, as he hasn't been active for quite a while now.

tlylt commented 2 years ago

A higher priority for me is to add support for more charts such as pie, line, bar, histogram etc.

For supporting basic charts, I am thinking we can support multiple javascript charting libraries like chart.js, in the same fashion as this markdown-it plugin:

within the markdown, write the data as json in a code block (language info is the chart library name):

\`\`\`chartjs
{
"type": "pie",
"data": {
"labels": [
  "Red",
  "Blue",
  "Yellow"
],
"datasets": [
  {
    "data": [
      300,
      50,
      100
    ],
    "backgroundColor": [
      "#FF6384",
      "#36A2EB",
      "#FFCE56"
    ],
    "hoverBackgroundColor": [
      "#FF6384",
      "#36A2EB",
      "#FFCE56"
    ]
  }
]
},
"options": {}
}
\`\`\`

transform it into the respective HTML code
inject the CDN script of the required charting library in the page

I will need to test out the plugin to see if we can use it as is or make our own with a similar workflow. Any comments or alternative plans on this are welcome :)

crphang commented 2 years ago

@tlylt I will not be working on it. Go ahead :muscle: !

ang-zeyu commented 2 years ago

The primary thing to keep in mind with this and #984 is author usability.

A lot of these external libraries require that you provide options as a json configuration object, from my very brief scan of chart.js this seems to be the case as well. This flexibility is great for general app development but may be a little clunky for markbind. (versus something like inline puml where we can assume users have puml knowledge) Not to say we can't expose the original configuration format to the author, there's also value in directly exposing it so that this is more maintainable; Abstracting everything into simpler interfaces is going to be costly.

E.g. for the data option, per the OP, we should at least consider transforming data from external sources easily, and not require that this data: [45, 25, 20, 10], be manually written. Default configurations could be provided / added as well for the most common use cases so the json configuration object need not (or at least, less of it) be manually written.

Ergo also to keep in mind: if you see any rather complex-to-use configuration options (that have use cases in markbind) that ought to be extracted, you could consider whether we need to abstract over it.

Also for data: https://markbind.org/userGuide/reusingContents.html#importing-variables-from-other-external-file-formats

We currently already have the above lower level ways of importing data from json / csv files. This could be linked up with the data option in chart.js easily.

<chart>
type: 'line',
data: {
  datasets: [{
    data: [
      {%for xx in ... %}{{xx}},{%endfor %}   // create the array and commas
    ]
  }]
}
</chart>

❗ ❗ From an author perspective, I'm not sure if this is enough or some abstraction should be done over this particular part (supplying data). For example, <line-chart src="foobar.csv">.

@damithc @tlylt

We can also consider starting out with this usage (i.e. the author needs to know how to supply the data option) then abstracting over it later.

write the data as json in a code block (language info is the chart library name):

https://markbind.org/devdocs/devGuide/writingPlugins.html#tag-behaviour

You could use this ^ to define your own "special tags". Its what we're using for inline <puml> tags currently, so that "conflicting syntax" (with html / md) can be inside.

It follows the same parsing rules as <script / style>, and being html tags you can provide configuration options as just simple attributes.

We should avoid mixing triple backticks syntax as its use case is well established for code blocks already.

damithc commented 2 years ago

❗ ❗ From an author perspective, I'm not sure if this is enough or some abstraction should be done over this particular part (supplying data). For example, <line-chart src="foobar.csv">.

@damithc @tlylt

We can also consider starting out with this usage (i.e. the author needs to know how to supply the data option) then abstracting over it later.

Fully agree. We can start with this kind of support first and add more user-friendly syntax for more common chart types later. i.e., it's in line with the principle 'make common things easy, make rare things possible'

tlylt commented 2 years ago

Recent update: GitHub now has native support for Mermaid: https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/

Perhaps it's mentioned somewhere but I can't really find it...why did we settle on creating custom HTML tags like \<puml> instead of putting it as part of the code fence? From an author's perspective writing mainly markdown:

code fence is a common markdown feature and the understanding of specifying a language after ``` is commonplace as well
whether is it puml or mermaid, they are actually lines of code so would be good to put them in the code fence for potential syntax highlighting as well as just easy of viewing, especially on a standard markdown renderer(e.g on GitHub/GitLab/VSCode interfaces)
writing ```puml is arguably more "smooth" than writing \<puml>
given the general support on major platforms like GitHub/GitLab, and potentially other places going forward, ease of migration will definitely be a selling point

damithc commented 2 years ago

@tlylt Isn't the purpose of fenced code feature is to show syntax-highlighted code? If we use the same syntax for embedding diagrams, how do you distinguish between the two features? But yes, given that neither puml nor mermaid has syntax highlighting support as of now, seems like it can work, and I'm OK to support the same for the sake of compatibility.

damithc commented 2 years ago

In any case, given GitHub already supports mermaid, I think we'd better start supporting it too :-p

tlylt commented 2 years ago

@tlylt Isn't the purpose of fenced code feature is to show syntax-highlighted code? If we use the same syntax for embedding diagrams, how do you distinguish between the two features? But yes, given that neither puml nor mermaid has syntax highlighting support as of now, seems like it can work, and I'm OK to support the same for the sake of compatibility.

hmm I see your point now... indeed if someone wants to just show the raw puml/mermaid code with syntax highlight support (may not be available at the moment), I guess there is no way to do it if we use code fence. The only workaround is to not specify the language info.

ang-zeyu commented 2 years ago

Relevant PR: https://github.com/MarkBind/markbind/pull/968

code fence is a common markdown feature and the understanding of specifying a language after ``` is commonplace as well

Yup, but for syntax highlighting.

At the time of this PR (2 years back) ^, ``mermaid was never really "standard".` syntax was our very first choice since it is more in line with MarkBind's components in general.

My personal 2 cents today as well still is that markdown code blocks shouldn't be used to generate anything other than code blocks, its mixing use cases and syntaxes that have nothing to do with each other semantically. Also, not quite commonmark compliant in that triple backticks/tildes should generate fenced code blocks.

writing ```puml is arguably more "smooth" than writing

Another smaller point raised (in the pr) was that specifying <puml> attributes/options (e.g. name) using markdown-it-attrs isn't quite as familiar / smooth as html attributes. (although, we do use it quite frequently for other code block features now)

In any case since github has started supporting this syntax, I agree we should support it as well (or at least the option of it). We can create a simple markdown plugin addition to replicate this functionality quite easily (i.e. transform ``puml to`). A plugin system addition to add markdown plugins from our plugins would be welcome first too though!

tlylt commented 2 years ago

Looking through some of the javascript charting libraries, I realised that it is currently possible to create charts by writing some HTML/javascript code in a .md file. This is because as stated in the UG:

A MarkBind source file can contain a mixture of HTML, JavaScript, and CSS as a normal web page would.

There is a caveat that for inline HTML elements, it will be wrapped with a <p> tag (which doesn't seem to cause much of an issue in general)

So for example, chartjs, the steps are:

Define a canvas to receive/render the chart
Import the charts library via a CDN
JS code to initialize and insert the chart into the canvas defined.

Output

So

'make common things easy, make rare things possible'

Seems like "rare" things are already possible

A plugin system addition to add markdown plugins from our plugins would be welcome first too though!

@ang-zeyu sorry I don't quite get what you mean by "markdown plugins", what's the difference between this and the current kind of plugins?

The primary thing to keep in mind with this and https://github.com/MarkBind/markbind/issues/984 is author usability. A lot of these external libraries require that you provide options as a json configuration object, from my very brief scan of chart.js this seems to be the case as well. This flexibility is great for general app development but may be a little clunky for markbind. (versus something like inline puml where we can assume users have puml knowledge) Not to say we can't expose the original configuration format to the author, there's also value in directly exposing it so that this is more maintainable; Abstracting everything into simpler interfaces is going to be costly.

I agree with the points mentioned. I think I am trying to be careful about what we decide to abstract such that the abstraction is useful, maintainable, and not leaky. Working on charts/visualizations in my other projects, I realized that people have varied needs & diverse requirements in terms of what a chart should look like for their use case(which is why charting libraries have a huge list of configurable options). Therefore, I would think that it is quite hard for someone to skip reading/referring to the official documentation of a charting library in order to find what's the correct syntax/options to use.

I guess at least for a start, this is what I propose:

Include documentation/examples on how it is possible to utilize some of these charting libraries
Extract out a few common chart types with minimalistic options to allow for really simple use cases.
- E.g. a way to create a piechart by doing something like <pie-chart>data = [1,2,3]</pie-chart>.
- The user need not know what is the underlying charting library in use, and the user is limited in terms of the options provided.
Create a general abstraction to simplify the typical steps required

Elaboration on pt 3:

import (help import the script according to the type defined)
define a location to place the chart (user define the data/option right here, or pass in from an external file via data attribute)
init with a data/option object (help pass the data/option and generate the required script to append to the page)

Example syntax:

<chart type="chartjs">
{
    type: 'bar',
    data: {
        labels: ['Red', 'Blue', 'Yellow', 'Green', 'Purple', 'Orange'],
        datasets: [{
            label: '# of Votes',
            data: [12, 19, 3, 5, 2, 3],
        }]
    },
    options: {
        scales: {
            y: {
                beginAtZero: true
            }
        }
    }
}
</chart>
<chart type="chartjs" data="/chart2.json"></chart>

ang-zeyu commented 2 years ago

@ang-zeyu sorry I don't quite get what you mean by "markdown plugins", what's the difference between this and the current kind of plugins?

an extension here, more specifically maybe something like this to allow our plugins to tap into markdown-it's own plugin system (for keeping everything isolated within markbind-plugin-plantuml.js as a plugin):

{
  markdownItPlugins: (md) => { ... }, // markdown-it-center-text.js for an example
}

tlylt commented 2 years ago

an extension here, more specifically maybe something like this to allow our plugins to tap into markdown-it's own plugin system (for keeping everything isolated within markbind-plugin-plantuml.js as a plugin):
{
  markdownItPlugins: (md) => { ... }, // markdown-it-center-text.js for an example
}

To confirm, do you mean to allow users to create a plugin that we pass the markdown content as input and the plugin can work on the transformation of the markdown content before we process the markdown (so somewhere after Prerender and before markdown rendering)? Something like:

module.exports = {
  processMarkdown: (pluginContext, md) => {

  },
  processNode: (pluginContext, node) => {
    if (node.attribs.id === 'my-div') {
      cheerio(node).append(pluginContext.content);
    }
  },
  postRender: (pluginContext, frontMatter, content) => {
    const $ = cheerio.load(content, { xmlMode: false });
    // Modify the page...
    $('#my-div').append(pluginContext.content);
    return $.html();
  },
};

(for keeping everything isolated within markbind-plugin-plantuml.js as a plugin):

Still don't quite get what you mean by isolating 😅. As in creating the functionality of the markbind-plugin-plantuml.js as a markdownIt plugin (compatible and usable for anybody using markdownIt) and then internally, the markbind-plugin-plantuml.js can plug the plugin into our markdownIt instance so that it alters how Markdown parsing is done?

ang-zeyu commented 2 years ago

To confirm, do you mean to allow users to create a plugin that we pass the markdown content as input and the plugin can work on the transformation of the markdown content before we process the markdown (so somewhere after Prerender and before markdown rendering)?

Still don't quite get what you mean by isolating 😅. As in creating the functionality of the markbind-plugin-plantuml.js as a markdownIt plugin (compatible and usable for anybody using markdownIt) and then internally, the markbind-plugin-plantuml.js can plug the plugin into our markdownIt instance so that it alters how Markdown parsing is done?

Nope, nothing so complex 😅. I'm referring to tapping into markdown-it's own plugin system, the same one you were exploring in https://github.com/MarkBind/markbind/pull/1685. Processing markdown twice also produces undefined behaviour.

Isolating as in for cohesion / code quality (a little odd for MarkBind's own plugin code to be littered all over its files)

{
  markdownItPlugins: (md) => { ... }, // markdown-it-center-text.js for an example
}

md is passed here to add/edit/update markdown-it rules in its plugin system. (see markdown-it-center-text.js for an example, near the bottom)

Just one syntax / suggestion to tap into their plugin system, feel free to explore others!

update: simple solution here https://github.com/MarkBind/markbind/pull/1940#discussion_r884140657

SPWwj commented 1 year ago

Recent update: GitHub now has native support for Mermaid: https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/

I can help to bring this feature into markbind

damithc commented 1 year ago

I can help to bring this feature into markbind

@SPWwj we are somewhat conflicted about this feature. On the one hand that's how Github implemented it, and it would be nice to be consistent with them. On the other hand, three back ticks followed by abc by right should be a code block with abc syntax highlighting, not a diagram.

SPWwj commented 1 year ago

@SPWwj we are somewhat conflicted about this feature. On the one hand that's how Github implemented it, and it would be nice to be consistent with them.

Agree, this syntax is most intuitive.

On the other hand, three back ticks followed by abc by right should be a code block with abc syntax highlighting, not a diagram.

However, I'm uncertain about the necessity of this limitation.

tlylt commented 1 year ago

On the other hand, three back ticks followed by abc by right should be a code block with abc syntax highlighting, not a diagram.

However, I'm uncertain about the necessity of this limitation.

Pls see the conversation above and answers given by Zeyu. https://github.com/MarkBind/markbind/issues/1663#issuecomment-1050850748

MarkBind / markbind

Support basic charts #1663

984

1079