Improve Chart update performance with many datasets

[jwedel]

Feature Proposal

We are developing a web application with a highly dynamic chart that is used to visually find anomalies in a set of hundreds and even thousands of manufacturing curves, sumtimes consisting of multiple Chart.js datasets.

This will end up in 10.000 dataset managed by chart.js.

We support hover highlighting of curves as well as zooming using the zoom plugin.

Starting at a few hundred dataset, the interactivity of the chart degrades significantly.

After doing intense performance analysis in the browser, it turned out that the redering itself is still very fast (10-20ms). However, the chart.update() takes up to 1s (!). When drilling down what actually is the problem, it turns out that there are two key problems:

1. The getDatasetMeta method which consumes most of the CPU time
2. The inability to do fine-grained updates on the Chart

On 1.) The biggest problem is the lookup of the metasets. Instead of doing an index access on the array, it is doing a Array.find. As this method is used in multiple places in the update inside loops over the datasets, this results in a complexity of O(n^2) which gets extremely slow with growing datasets.

Here you can see that the update take >600ms:

When zooming in, you can see that each call of getDatasetMeta takes in this example takes 2.7ms which does not sound like a lot. But this is called n*n times, so with 1000 dataset, it could be called a million times worst-case (on average it's less as find() will stop on a hit). Nevertheless, this adds up to the 600ms on each update() which is way too laggy for any user interaction.

On 2.) The 'update()' method does a lot, and especially it does a lot of change detection. The problem is, when it comes to performance optimization, I as the developer know what I have changed and I only want to update this. Also in when zooming, the plugin always calls 'update()' which, as explained also due to 1.) gets extremely slow with growing number of datasets.

One example: When hovering over a curve, I want to change the line thickness to 3 (bold) and all other datasets colors to grey (fade out) and change the order property so that the curve is on top of the stack. When I do this, I only want to update the metasets and I know that I did not change the points, the datasets itself (as in adding / removing), I did not change the scales, controller etc... Currently, AFAIK, this cannot be done in Chart.js.

Possible Implementation

On 1.) To fix one, I monkey patched the getDatasetMeta to use index access. I looked at all the code and I can't find a problem and it seems to work and it is very fast.

this.chart.getDatasetMeta = (datasetIndex) => {
      const dataset = this.chart.data.datasets[datasetIndex];
      const metasets = this.chart['_metasets'];
      let meta = metasets[datasetIndex];
      if (!meta) {
        meta = {
          type: null,
          data: [],
          dataset: null,
          controller: null,
          hidden: null,
          xAxisID: null,
          yAxisID: null,
          order: (dataset && dataset.order) || 0,
          index: datasetIndex,
          _dataset: dataset,
          _parsed: [],
          _sorted: false,
        };
        metasets[datasetIndex] = meta;
      }
      return meta;
    };

Ther are the _sortedMetasets which respect the order property so there is no need to have any other order in the _metasets array.

on 2.)

When for example only wanting to update the colors + ordering, I'll call private apis of Chart.js:

this.chart['_updateDatasets']();
        // this.chart.buildOrUpdateControllers();
        // this.chart['_updateMetasets']();
        this.chart['_sortedMetasets'] = this.chart['_metasets']
          .slice(0)
          .sort(this.compare2Level('order', 'index'));
        this.chart.render();

As this solution is not hacky and not very future-proof, I would love to see this implemented in Chart.js,.

So 1. could be just changed in my option, for 2. there should be a partial update. Something like:

chart.update({
   pointsChanged: false,
   datasetsChanged: false,
   datasetMetaChanged: true,
   orderChanged: true,
   ...
});

When this object is not passed, it would just do a full update as always.

I am happy to contribute with some guidance, if this would help.