Server side rendering

[autarc]

I am searching quite a while for a library which provides the capability of universal chart editing without requiring the DOM itself. All other implementations (Canvas, CSS or SVG) are tied to the browser - using the built in abstraction for defining new node and compose them. One advantage of vector graphics like SVG is the underlying representation as a string. A server can take advantage and render such basic string version directly or create a static image bitmap as a fallback. Originally I planned to use react for this approach, but I guess it could be beneficial for different use cases and therefore fit better into a general chart tool. The svg-builder already covers the basic construction of the elements.

What do you think ? If you agree, I like to take a look into it. Be aware that it will internally apply a larger restructuring of the existing code, but the public API should remain and just get extended.

[gionkunz]

Hey there again :-) I like the idea of rendering Chartist server side! I actually researched myself for some time and after reading this article http://www.smashingmagazine.com/2014/05/26/love-generating-svg-javascript-move-to-server/ I was really curious about this and I'd love to make this work somehow.

Some thoughts

1. Generally Chartist was developed to work in the browser and therefore It's one of it's main goals to leverage clean separation of concerns in the browsers. All SVG styles that can be styled with CSS should be styled with CSS. Also the chart types were optimized to gain some of their customization by using techniques that allow style customization only by CSS (like bar charts are done with lines, pie charts too to allow playing with stroke-width)
2. I maintain my own SVG library (Chartist.Svg) which currently is of a DOM manipulation nature and creates / modifies DOM rather than the creation of a virtual object model that could be reflected to DOM or to an SVG XML string (or whatever else). This is also very browser oriented.
3. Chartist has a lot of features that are optimized for browsers (like responsive configuration overrides) and this is also what the core concern of Chartist is (simple responsive charts). Developing Chartist for server side should be a feature rather than a core concern.
4. I believe, looking at 2, this could be solved with a virtual DOM like https://github.com/tmpvar/jsdom or https://github.com/Matt-Esch/vdom. However, looking at 1, I think this would need to be done differently and styles would need to be transformed from CSS to SVG attributes. Then using a virtual DOM in node, the DOM can be stringified to an SVG XML string easily and rendered even further with https://www.npmjs.org/package/svg2png for example.
5. All of the above is very much just a surgery process to hack on Chartist as a pure browser library until it can be used server side / DOMless. The real solution, which I believe you had in mind, would be to abstract away the rendering completely and let adapters decide if it should go to DOM or to SVG, PNG or whatever else.
6. Considering all of the above and if we would AIM for 4 then Chartist.Svg would probably be the place where to start and try to abstract away rendering so that there will be a DOM adapter as well as a SVG string adapter. However, this would not help with the issue described in 1. Not having a CSSOM available to extract relevant styles an convert them to SVG attributes could be a challenging thing right. Maybe this could be solved by adding general support for overriding CSS styles of chart elements via the style attribute and some configuration in javascript (which I really wanted to avoid) would help here. So we could again use Chartist.Svg to parse style attribute changes and reflect them as SVG attributes when used in SVG string mode.

My overall feeling is that Chartist.js should stay as is (optimized for browsers) but there should be a module / plugin / wrapper that uses virtual DOM / imagemagic / https://github.com/walling/node-rsvg to render images.

What are you thoughts?

Cheers Gion

[autarc]

Yeah, the idea is to define the actual SVG templates under the hood purely as a string and then attach them to the DOM. Aside of this direct browser embed, an additional method could refer to only render the markup which inlines the previous declared styles (DOM attachement would instead refer to the appropriate CSS classes). The building itself happens dynamically and will be similar to the current approach - although this wouldn't require the context of the browser and allows to render it in other environments like the server or even web workers. As a client side library Chartist should never expect a specific server component or 3rd party plug-in, using a jsdom or PhantomJS adapter always feels awkard in such situations. Not relying on the DOM has also the advantage of improved generation of complex structures, as the creation happens in memory and the animations are mainly happening in either SMIL or CSS. Providing other formats to convert the charts is probably unnecessary. If people really need an alternative rendering mode for older browsers d3 or rapahel should work fine for them. Offering a solution for modern browser with string based DOM support is probably enough for the future.

Should I give it a try ? What are your thoughts :)

[gionkunz]

Yeah, the idea is to define the actual SVG templates under the hood purely as a string and then attach them to the DOM.

Well I'm somewhat supportive, but only if we create this in a browser friendly manner. I don't think it would be a good idea to render a DOM string and attach it to the container DOM for performance reasons. On data or config updates as well as on window re-size, we would need to recreate the whole DOM again.

As recreating the whole DOM is currently happening, and I'm in the process of changing that, I think I need to share my thoughts on a different issue here. To solve #3, #37 and some overall performance gains, I'm currently working on a core refactoring already. This included the following ideas:

• Split the rendering of the DOM into phases / layers, where each layer will considered valid after rendering and subsequent renderings will skip layers that are valid
• If there is an update on the data, this could result in a whole layer being invalidated (like adding new data) or just parts of the layer being invalid (like modifying just some data points in a chart). The background grid is also on a separate layer, where a data update could potentially result in a new grid which would invalidate the whole layer.

This way I was hoping to enhance performance of progressive data updates and also by splitting the data grid from the chart, and through polymorphism, it would be possible to create a compound chart that shares the grid layer but has N chart plot layers coming from individual charts that are combined into one single one. I did already commit one core method that helps me identify deltas for this kind of process: https://github.com/gionkunz/chartist-js/blob/develop/source/scripts/chartist.core.js#L660-L722

Originally my intention was to run this function on config changes and on the data objects containing series and labels etc. The function creates a delta descriptor object that describes the amount and nature of deltas for nested objects and arrays. This way it would be possible to selectively invalidate the rendered components and isolate the DOM updates.

Thinking about all this again, if we start to think of the SVG DOM more as a virtual model to create real DOM or just an SVG string I believe this could also be a better approach to solve #3.

The idea would be something like this:

• Chartist.Svg creates a virtual DOM rather than real DOM using something like Chartist.Vdom
• Chartist continues to perform full recreation cycles on re-size, data updates and config updates but will not recreate the real DOM (which is almost 80% of compution time) but the virtual DOM.
• It's the responsibility of Chartist.Vdom to detect deltas after a full VDOM recreation and reflect them to the target system (either the real DOM or the SVG document).
• Chartist.Vdom would recycle the real DOM as much as possible with optimizations based on delta analysis / diff algorithm. This would need to be performed on element, attributes and probably also considering some metrics for child elements. Also this might use something like levenshtein distance with a threshold in order to detect modifications vs. delete + add. However, here I'm very open for suggestions. Maybe we can also look at how FB React does this, as I believe they are dealing with something similar for HTML DOM.
• Chartist.Vdom would only update the deltas found and the DOM operations would be limited to the smallest amount possible
• Chartist.Vdom would not necessarily need to store HTML DOM like structures but could also use a more binary representation for later optimizations / faster processing
• Chartist.Vdom could also use different adapters that render to a Canvas, Images or SVG document to create SVG.

[gionkunz]

Also, I'd like to avoid 3rd party dependencies as far as the browser dependencies go. So for https://github.com/JoeChapman/svg-builder we should stick to Chartist.Svg and do what we need to change there.

[gionkunz]

As you can see from this jsperf http://jsperf.com/dom-vs-innerhtml-based-templating/521 it looks like giving control to the browser how to create the DOM by putting a string to innerHTML is almost double as fast as using DOM manipulation API. However, with innerHTML you'd need to reconstruct the whole DOM every time.

So if we would change Chartist.Svg to build a string and replace the container innerHTML it would be almost double as fast. However, I think having a diff analysis and only updating parts of the DOM by using a virtual DOM is still the best we can do.

[autarc]

Thats what I had in mind: using the advantage of a static representation for the initial rendering at client and server, with onging enhancements done by the browser afterwards. If you prefer not to include an existing implementation for a builder we could re-do it - althought it will in the end consist of different modules (e,g, for converting CSS styles as inlines) anyways.

[gionkunz]

Okay, as this is a fairly heavy change to the core I suggest we collaborate on the changes. As soon as you have an initial commit, let's create a feature branch where we can collaborate. How does that sound?

[autarc]

Sounds good. I am changing the configuration of the serve task you mentioned and will start with an implementation of the virtual builder with the next release.

[gionkunz]

Cool :-) Looking at https://github.com/Matt-Esch/virtual-dom (used by react) I guess we can adopt a lot of stuff from there. Using https://github.com/Matt-Esch/virtual-dom as a dependency would also be a solution but I just think it would be too much for chartist and we should produce a light version of an virtual SVG dom representation. Looking at https://github.com/Matt-Esch/virtual-dom/blob/master/vtree/diff.js could be beneficial too I guess.

[gionkunz]

Once you update the PR in #86 I'll merge it locally and push to develop.

[gionkunz]

This is also nice to read: http://calendar.perfplanet.com/2013/diff/

[autarc]

Will be useful for the creation of the DOM patches. Here is also an overview about different libraries supporting virtual abstractions. The good thing is that our implementation will be specifically focused on SVG - so the actual elements and attributes are more restricted.

[gionkunz]

The question really is, do we need to restrict it to SVG? The semantics of DOM elements, if HTML, SVG or even custom elements / custom attributes as part of the web components specs doesn't really matter right? They all consist of element name, attributes and namespace attributes. Nothing really more. I don't see a reason why we would need to validate what's getting created.

Also we need to be aware that inside a foreignObject you would want to create anything that your browser supports, being HTML, MathML or anything else.

[gionkunz]

Wow the dart vdom shows incredible performance! Amazing. Let's start somewhere and work on this together.

[backflip]

This sounds promising! One small correction: virtual-dom is not used by React but rather "heavily inspired" by it (see README).

[gionkunz]

You're right Thomas. I thought they made their virtual DOM stuff into an independent module. I would really love to make my own virtual DOM implementation optimized for SVG :-) This would be a fantastic performance exercise.

[twwwt]

What is the current state of this issue? Is there a way to render Chartist.js charts server-side (into SVG or another format such as PNG)?

[autarc]

Unfortunately at the moment its still not possible to render the SVG charts at the server. Aside of creating a virtual-DOM renderer, you could also setup a headless system (e.g. via Phantom to make screenshots and save them.

[gionkunz]

You might checkout this https://github.com/panosoft/node-chartist

[alexgig]

@twwwt @Autarc

As @gionkunz pointed out, check out node-chartist. It allows you to create SVG charts using Chartist on the server (no Phantom or virtual-dom needed).

Hope it serves you well!

Best, Alex