RFC: creating an enhanced version of LuCI statistics graphs?

[hnyman]

Creating an alternative version of LuCI statistics with enhanced graphics?

We are currently using the year 2003 version of rrdtool 1.0.50 in LuCI statistics. The benefit is that the RRD library 1.0.50 is really small, compact and has no external dependencies. It is really efficient on flash space.

The drawback is that we miss all the developments for RRDtool in the last 19 years and the graphs are rather basic and show their age. In practice there is also no support for the current RRDtool for other needs than LuCI statistics.

The graphs from the current RRDtool 1.8.0 are more polished and crisp. The 1.8.0 is also aware of the time_t size and can adjust to 64bit time in 32bit systems.

One year ago I briefly tested with new RRDtool, when looking at the musl 64bit time_t issues. See

• rejected proposal to accept 64bit time_t and modify the ancient RRDtool for that: https://github.com/openwrt/packages/pull/16703
• current modification of keeping 32 bit time in 32 bit CPUs: https://github.com/openwrt/packages/pull/16835

We ended up patching the ancient RRDtool version to use 32bit time with 32bit CPUs despite musl always has 64bit time_t. That is a short-term kludge, but offers no compatibility with the current RRDtool versions and is not future-proof. (However, that enables the old databases to be compatible with the current musl. That is useful to those (few) that have preserved stats data in sysupgrades.)

Triggered by the recent forum discussion in https://forum.openwrt.org/t/update-rrdtool-up-to-1-4-or-higher/139473 I have now implemented a new upgrade to RRDtool 1.8.0, and I am considering the possible roadmap scenarios for LuCI stats.

The crucial aspect of RRDtool 1.8.0 is that RRDtool has in 2005-2008 moved from the ancient self-contained lib to more complex graphics libraries cairo, pango, freetype etc. (requiring also glib2). So, the size of the required libraries is approx. 3 MB larger (as measured from a compressed sysupgrade image for RT3200). That is a lot of flash space :-(

Due to the size increase, I see no feasibility in wholly replacing the old rrdtool 1.0.50 with the new version.

Instead, there might be two alternative versions of rrdtool and LuCI statistics.

For that purpose, I have created the patch set that

https://github.com/hnyman/packages/tree/rrdtool180

• creates the necessary graphics libs in feeds/packages: harfbuzz, fribidi, pango, cairo
• creates an alternative new "rrdtool" next to the old "rrdtool1"
• modifies collectd to select between the old rrdtool 1.0.50 and the new 1.8.0 at build time

https://github.com/hnyman/luci/tree/rrdtool180

• modifies LuCI statistics to select between rrdtool 1.0.50 and 1.8.0

(Both luci-app-statistics and collectd need to have the enhanced version selected manually, so this is just a proof-of-concept at this point. But it works.)

The non-standard video feed contains versions of pango, cairo, harfbuzz and fribidi authored by @dangowrt , but they have too many features enabled (and caused compile errors), so I created new minimal versions of the same packages into the packages feed.

So, regarding a possible future roadmap I am considering:

• renaming the old rrdtool1 as rrdtool-mini (or rrdtool-old ???)
• renaming the old collectd-mod-rrdtool to collect-mod-rrdtool-mini (or -old ???)
• introducing "-mini" versions of pango, cairo, harfbuzz and fribidi into the packages feed. The video feed might have more full version of them.
• introducing new rrdtool (the current 1.8.0) (and also keep the old rrdtool1 as rrdtool-mini)
• introducing new collectd-mod-rrdtool that links to the new rrdtool 1.8.0. That requires patching collectd to have a two rrdtool plugins. Having two separate collectd-mod-rrdtool variants would enable LuCI to have one master option about the normal or enhanced LuCI stats, instead of separately defining the underlying lib also for collectd-mod-rrdtool at the build time.
• modifying LuCI to either
  • have a similar stats build-time config option that I use now, or alternatively
  • offer two statistics variants, "normal" and "enhanced"

Sounds feasible?

Specific questions:

• I have now been testing with RT3200 that has 64bit CPU, so the RRD database is compatible between 1.0.50 and 1.8.0. I do not see much point in patching the 1.8.0 so that it would always still use 32bit time with 32bit CPUs. I would prefer to keep it in sync with the upstream RRDtool. So I propose that the new 1.8.0 would be the current default without patching it to use 32bit time. That would more easily enable it to be used for other purposes than just LuCI stats. (It would break compatibility to old 32bit databases, but those can be manually converted. I tested that a year ago) Ok?
• Does introducing -mini versions of pango, cairo etc. in feeds/packages sound feasible?
• Renaming rrdtool1 to rrdtool-mini?

@jow- @dangowrt

[hnyman]

Some screenshots for comparison.

E.g the grid is more muted in 1.8.0 and in the year-long graph it shows both the weekly grid and months

[jow-]

The dependencies for modern rrdtool are borderline insane imho. On my hidpi screen, the 1.8.0 examples just look blurrier than the old ones - nothing compelling to warrant the update.

[hnyman]

The new RRDtool's dependencies are really extensive, sure. That is the major drawback from our perspective. And the main reason why we have not done anything during the years.

Thatswhy I am not rushing to anything, but just wanting to collect opinions about the proposal.

My thinking is more about offering the current version of RRDtool, and then along that a new non-default variant of the LuCI stats. Even if we do not modify the LuCI stats, offering a current version of RRDtool would require this large new library set and modification of collectd etc.

But yeah, the actual benefit from new stats graph versions is probably rather minimal for most casual users.

Ps. I see the graphs on a 4k desktop 27" display, and the new charts look nice and crisp. (But the fonts may sometimes look a bit blurry. Might be a screenshot effect of me having set Windows to scale the display somewhat and Firefox to additionally scale a bit more. I have also not done anything to the font aliasing options.)

[systemcrash]

You've evidently put quite a bit of thought into this. New rrd fixes everything in one go, which is a strong motivator.

Since the topic is graphing... I've worked on and off with two libraries:

• d3.js
• Vega-lite

over the years, and find them really attractive: both the amount of visualization types available, and how small the library is. Graphs one can zoom and interact with. These will certainly give you something enhanced.

Having d3 and/or vega available so that 'graphing' LuCI apps can be converted seems a viable long-term alternative.

D3 is pure web standards based. The latest D3 is available in all CDNs, and the lib is anyhow small enough for flash. 7.7 minified to 270KB.

https://github.com/d3/d3 https://d3js.org/

A similar story for vega-lite, outputs to SVG, which all browsers support anyway, and is pure web standards based. Lite is similar, and weighs in at 270KB minified. Feed vega-lite with JSON, CSV or raw values, and you're done. (Vega is lower level and weighs 500KB).

https://vega.github.io/vega-lite/

The only drawbacks I can think of now are that they require JS to be on in the browser, which a minority of power users may not have.

[CAFxX]

A potential alternative could also be using something like https://javascriptrrd.sourceforge.net/ and have the parsing/rendering happen fully in the browser. The realtime graphs already use this approach (but different implementation) anyway.

[systemcrash]

I wanted to get your opinions on live loading scripts from a CDN from within apps and protos, so we can offload (optionally dependent) scripts from a package to keep things small. E.g. an optional bar-chart is required from a remote https resource.

fetch('http://example.com/resource.js')
  .then(response => response.text())
  .then(scriptCode => {
    // You can evaluate the script code or use it as needed
    eval(scriptCode);
  })
  .catch(error => {
    console.error('Error loading resource:', error);
  });

[jow-]

IMHO, LuCI should remain self contained, no loading of external resources.