Reduce “binary” size

[Hywan]

Hello :-),

Is there a way to reduce “binary” (viz.js) size? For instance, PNG and xdot outputs are not interesting for me. Can I remove them? Maybe if I can compile dot without these renderers, I can use this Makefile the same way?

[mdaines]

I've been looking at this recently. A recent commit 119049b0de8b319492f86ef34e1ade4af8d2efde does reduce the size a little, but removing renderers and other features may provide more savings.

Do you have a target size in mind? (And what features could you not do without?)

[vmarkovtsev]

Graphviz code isn't an example of good modular arch, neither is it's build system. It is hard to cut out parts. Hard, but possible fo sure.

[Hywan]

My goal would be to get only one renderer, like dot and being able to export to SVG only. Is it possible?

[mstefaniuk]

I would select xdot only renderer.

[Hywan]

@mstefaniuk Why?

[mstefaniuk]

In my project I'm using xdot output but not svg. But I've commented on this case to show different needs. To balance between them it would be nice to separate engine from renderers. They can be combined at runtime or dynamically loaded using requirejs.

[Hywan]

@mstefaniuk I guess enabling them at runtime is very difficult. But statically, this is maybe more possible. Toughts @mdaines?

[Hywan]

(How can we help?)

[mdaines]

@vmarkovtsev is right, the Graphviz build system doesn't provide all that many options. If you don't need neato layout, I guess this output from ./configure is not so encouraging:

plugin libraries:
  dot_layout:    Yes (always enabled)
  neato_layout:  Yes (always enabled)

One way to reduce the size of Viz.js (while also reducing features) would be to not include Expat. That's the sort of thing I would look at initially — compiler or build configuration settings. I do prefer the simplicity of one JavaScript file, however.

Does anyone have other ideas?

[Hywan]

@mdaines What about contributing to Graphviz in order to disable some layouts and output formatters?

[ghost]

@Hywan That would be the ultimate win as it would improve GraphViz and therefore viz.js as well.

viz.js is very large at this point and not much space can be won with minifying it, so if we can maybe split various options to allow selecting which options you need at build time creating the smallest js file needed.

The other option is to find a way to create multiple javascript files with each file giving only certain capabilities, (this might cause too much duplication however)

[Hywan]

@aquaalex Contributing to GraphViz is our best option. Is there someone with a good karma over there?

[magneticnorth]

Hi - we generally welcome all reasonable contributions.

I apologize a little for the build setup, but it's a product of its life and times. I could write a book.

In principle, the layout engines and drivers are almost completely pluggable. I don't think any of the layout engines are that big though neato probably accumulated a bunch of code when the stress majorization solver came in, and more when the directed variant (IPSEPCOLA - wasn't this supposed to be PEPSICOLA? too subtle for me) came in. You can un-ifdef the IPSEPCOLA code, probably.

There is no need to bring in the cairopango driver (or X11, or quartz, or even libgd) unless you want it. There is no need for ghostscript or expat either unless you expect to load external shape files. In that case, we use shape readers+renderers for cross-renderer compatibility and to get sizes before layout. In any case, getting size of arbitrary text requires having font binding and font rendering services (like fontconfig and freetype/pango/quartz). If none of these are available at ./configure time, there is some fallback code (hardwired raster fonts) but they can look ugly and just a few basic fonts are supported. (At one point we had fixed font tables for 10 or 20 Postscript fonts like Palatino listed in graphviz/lib/common/ps_font_equiv.h but I think that code is gone - obviously dynamic services can do a better job, and resurrecting this from git (migrated from subversion or mercurial) seems nontrivial.

Can someone provide numbers showing where the viz.js code bloat comes from?

Thank you for caring about Graphviz.

Stephen North north@graphviz.org

[adamserafini]

Interestingly, the website www.webgraphviz.com uses a build of viz.js with a size of only 1.36 Megabytes.

This appears to be smaller than any 'official' release of viz.js. Perhaps it's an earlier version? Unfortunately I can't see any version information in their build.

Does anyone know how they got a build that small? It appears to fulfil the basic functionality I need viz.js for (data structure visualisation on the web) so I was thinking of 'stealing' it from their website. A reproducible build of that version would be better though.

[japgolly]

I just started playing around with this again.

I only need DOT input and SVG output and long ago I used that to reduce the size. I've just now updated it - if you're interested, you can see two branches on my fork.

Basically, I've gotten it down to as low as 1.1mb.

GraphViz 2.28:

-rw-r--r--  1 golly users  1221072 Feb 24 15:52 viz.js
-rw-r--r--  1 golly users  1128359 Feb 24 15:52 viz-z3.js

GraphViz 2.32:

-rw-r--r--  1 golly users  1616625 Feb 24 15:46 viz.js
-rw-r--r--  1 golly users  1464333 Feb 24 15:45 viz-z3.js

viz-z3.js is smaller because it was compiled with -Oz -O3. I haven't tested this yet so I don't know how stable the z3 build is, but the viz.js files were compiled with optimisation flags that were stable back in 2013.

[japgolly]

(Btw: to make it clear, don't depend on any of that stuff without testing it. I'm not sure what does and doesn't work anymore. I haven't tested it yet myself.)

[Hywan]

Is it even possible to automate the testing? Good job so far! How is the next step?

[japgolly]

How is the next step?

Well 2.28 doesn't work, I tried making similar changes to it as I did to 2.32 to get it working and switch it over to cgraph but it just crashes at runtime. As for the 2.32 side, I curious to see if 2+ years of emscripten dev would magically reduce my custom build - it didn't. The file I generated years ago is 1,423,752 which is smaller. I may have screwed something up - I don't know - but I've concluded I'm going to stop playing around with this now and just stick to my old JS. Hopefully this will give someone else an idea to do better than me :)

[jaime-olivares]

There is a huge array at the beginning of the compiled js file. Something like this:

allocate([0,32,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,32,193,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,82,64,0,0,0,0,0,0,82,64,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,82,64,0,0,
0,0,0,0,82,64,0,0,0,0,0,0,240,63,0,0,0,0,0,0,…

…58,97,108,110,117,109,58,93,95,93,0,91,91,58,115,112,97,99,101,58,93,93,0,91,94,91,58,115,
112,97,99,101,58,93,93,0,91,91,58,100,105,103,105,116,58,93,93,0,91,94,91,58,100,105,103,
105,116,58,93,93,0],"i8",ALLOC_NONE,Runtime.GLOBAL_BASE+192848)

It takes more than 500Kb. I estimate it would take half of its size if stored as a long base64 string, reducing the overall file size in ~7%.

I know this JS file is autogenerated, but some hand manipulation may contribute to optimize the size of it.

[magneticnorth]

Well, better/worse, I’m sure the whole thing was generated by an algorithm in the first place.

Contact Yifan Hu yifanh on G m a i l and ask him for the code?

Stephen North

On Jul 23, 2016, at 11:09 PM, Jaime Olivares notifications@github.com wrote:

There is a huge array at the beginning of the compiled js file. Something like this:

allocate([0,32,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,32,193,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,82,64,0,0,0,0,0,0,82,64,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,82,64,0,0, 0,0,0,0,82,64,0,0,0,0,0,0,240,63,0,0,0,0,0,0,…

…58,97,108,110,117,109,58,93,95,93,0,91,91,58,115,112,97,99,101,58,93,93,0,91,94,91,58,115, 112,97,99,101,58,93,93,0,91,91,58,100,105,103,105,116,58,93,93,0,91,94,91,58,100,105,103, 105,116,58,93,93,0],"i8",ALLOC_NONE,Runtime.GLOBAL_BASE+192848) It takes more than 500Kb. I estimate it would take half of its size if stored as a long base64 string, reducing the overall file size in ~7%.

I know this JS file is autogenerated, but some hand manipulation may contribute to optimize the size of it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mdaines/viz.js/issues/52#issuecomment-234754347, or mute the thread https://github.com/notifications/unsubscribe-auth/ACtWz97qrhLQ8_N26pdYj0FcYVIlzKdVks5qYtd5gaJpZM4Gj2bi.

[jaime-olivares]

@japgolly, it would be great if you could publish your fork to NPM, so it can be easily consumed.

[jackycute]

I see the viz.js on http://www.webgraphviz.com/ again. It's only 312 kb, it's crazy, we should ask them how they did it.

[mdaines]

webgraphviz.com appears to be using an older build which is about ~1.3MB. It doesn't, for example, include Expat for HTML label support.

In terms of features, it seems like a smaller build should look something like this:

• Only needs to include the dot layout engine
• Only needs SVG output
• Doesn't need Expat

[DeBAAT]

That would exactly be the features I need for my purpose: the WP-GraphViz plugin for WordPress. Would it be possible to include this in the release download?

[mdaines]

I'm working on this in the viz-lite branch. I have removed Expat and the -lgvplugin_neato_layout option, which brings viz.js down to 2.2MB (458K gzipped) so far. I'm going to look at an approach similar to what @japgolly was working on as well.

[Hywan]

Excellent :+1: and bravo!

[mdaines]

By ensuring that all parts of the build use the -Os setting, I've reduced the size of the regular viz.js file to 2.3MB and viz-lite.js to 1.6MB. The gzipped sizes (obtained from running gzip with default settings) are 560K and 387K respectively. viz-lite.js will be in the next release (1.5.0). I welcome PRs for further size reductions, but so far this is the least "invasive" solution I can find.

[Hywan]

\o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/ \o/

Thanks! Incredible works, thanks!

[mstefaniuk]

Is it possible to add xdot output in lite version? Maybe it is not size enlarging.

[mdaines]

xdot output is included in the lite version. It only omits Expat and neato layout for now.

[Hywan]

@mdaines It sounds pretty correct.

[mstefaniuk]

Lite version is still not a part of bower package distribution. Please add it there.

[mdaines]

I will include viz-lite.js in the Bower package in the next release.

I'm curious to hear from anyone who uses the npm package. Does it make sense to include viz-lite.js there as well? How should that be done?