Brush & Zoom with >15k datapoints

[ux-engineer]

I've made a chart based on this demo: http://bl.ocks.org/mbostock/34f08d5e11952a80609169b7917d4172

That incorporates downsampling of up to 4 million datapoints in two line series'. After zoom end event the data must be resampled and reloaded to paths.

I noticed that whenever the line has more than 15 000 datapoints such line disappears from the canvas. When zooming in, after downsampling the bucketsize decreases and datapoints count increases when zooming in. How could I overcome this problem / restriction?

In this use case the user specifically needs to be able to zoom in to look at the data at the scale of 5 seconds per full (responsive) window width. One second has 10 points in one line and 20 points in the other. This use case has been traditionally done on the desktop app, but we are trying to achieve this in the browser instead.

We have already a Highstock implementation with downsampling which does work 'ok' without unbearable lag (often loads programmatic scaling in just a sec [pun unintended]). But if D3 could do the same or better, that would be great. As there also has been some limitations with the Highstock's customizability in this case.

I haven't yet come about any large dataset demos with this high amount of data along with downsampling (and then combined with zoom/brush and responsive design)... I got downsampling to work with responsive design...and now zoom/brush with downsampling, but encountered this when reloading data after resampling.

Lazyloading might not be an option, as the initial loadtime of this full dataset may be 20-40 seconds (well, depends on the Internet connection), size being 3 megabytes without timestamps (datapoints in constant frequency).

[mbostock]

Please use Stack Overflow tag d3.js to ask for help. Although I make an effort to assist everyone that asks, I am not always available to provide help promptly or directly. Stack Overflow provides a better collaborative forum for self-help: tens of thousands of D3-related questions have already been asked there, and some answered questions may be relevant to you.

When asking for help, please include a link to a live example that demonstrates the issue, preferably on bl.ocks.org. It is often impossible to debug from code snippets alone. Isolate the issue and reduce your code as much as possible before asking for help. The less code you post, the easier it is for someone to debug, and the more likely you are to get a helpful response.

If you have a question about D3’s behavior and want to discuss it with other users, also consider the d3-js Google Group or joining the d3-js Slack.

Thank you! 🤗

[ux-engineer]

Thanks for your answer @mbostock. However don't you think that having > 15k datapoints should still work with brush/zoom? Any idea what this might be about?

It was working fine before implementing brush/zoom, I even got 4 million datapoints drawn at the same time (not usable due lag with scroll though).

[mbostock]

I can’t think of any reason why the number of data points would have any effect on brushing or zooming, other than standard implication on browser performance. This zoomable area chart has about 8,000 data points and seems to do fine:

https://bl.ocks.org/mbostock/4015254

I do notice a few rendering glitches when it is embedded in an iframe, but this has nothing to do with D3 and would be a browser rendering bug that should be filed with the appropriate vendor. You could also switch to rendering the area in Canvas if desired.

To improve performance with larger datasets, the recommended strategy is to aggregate to reduce the number of data points.

In any case, please follow my advice above on asking for help on Stack Overflow. And I recommend trying to be more specific about what you want help with, and linking to example code on bl.ocks.org to serve as a demonstration for your question.

Also see Reporting an Issue for filing a D3 issue, but this doesn’t appear to be a D3 issue.

[ux-engineer]

@mbostock I'm sorry to bother you with this, but I've replicated this issue as based on your blocks Brush & Zoom. I made modifications to draw lines instead of areas, with random data generation: https://jsfiddle.net/envisio/d78hvjb5/12/

When using datapointsCount 15 000 the focus and context lines are both drawn. With 16 000 just the focus line gets drawn. And with 16 300 both are missing?

So in my mind there is a possibility for this to be zoom or brush related issue, as 16k datapoints is not that many yet for the browser to handle – or is it?

[ux-engineer]

Also D3 without brush/zoom features did handle drawing 3-4 million datapoints total with two lines just fine, although that would't be a practical solution.

[mbostock]

It appears to be your use of shape-rendering: crispEdges.

Again, please use Stack Overflow tag d3.js to ask for help. Although I make an effort to assist everyone that asks, I am not always available to provide help promptly or directly. Stack Overflow provides a better collaborative forum for self-help: tens of thousands of D3-related questions have already been asked there, and some answered questions may be relevant to you.

[ux-engineer]

That was it, many thanks!