search

htm-community / nupic.visualizations

Web application for interactive graphs, anomaly highlighting and online monitoring.
MIT License
17 stars 11 forks source link

Rendering a large file (200k+) takes too long - rangeSelect, pan, ... #16

Closed breznak closed 8 years ago

breznak commented 8 years ago

Working on real-life data, my files have something like 3M lines, even 280k lines file takes too long to open,zoom,pan, etc that it is not suitable for normal workflow.

Can you replicate the problem, and if so, what can we do to overcome it? Optimizing the framework, or need to switch to some faster components? I was unable to debug which part is "freezing" the system, but assume it's the dygraph as the problem appeared when zooming a graph.

jefffohl commented 8 years ago

We could perhaps use a web worker to parse the data - so this part is working on a different thread than the main. Then, we could chunk the data as the web worker processes the data, so you don't have to sit there waiting for the entire data set to be parsed. Just thinking out loud here, I can do some tests later - don't have much time today.

breznak commented 8 years ago

My recent idea was to use subsampling:

Maybe the framework could to that already?

breznak commented 8 years ago

@rhyolight in RiverView you may hit a similar problem, have you had to deal with it?

rhyolight commented 8 years ago

I limited the max # of rows to display in River View to 5000 I think.

Matt Taylor OS Community Flag-Bearer Numenta

On Mon, Dec 7, 2015 at 5:41 AM, breznak notifications@github.com wrote:

@rhyolight https://github.com/rhyolight in RiverView you may hit a similar problem, have you had to deal with it?

— Reply to this email directly or view it on GitHub https://github.com/nupic-community/nupic.visualizations/issues/16#issuecomment-162527222 .

breznak commented 8 years ago

We should try with the newest release, promises some speedup in fillGraph https://github.com/danvk/dygraphs/releases If it does not help, I'll contact upstream in https://github.com/danvk/dygraphs/issues/496

breznak commented 8 years ago

Updating the title to focus only on the problem of rendering - when rangeSelect, zoom at would take too much time.

adamborowski commented 8 years ago

@breznak I develop such framework: see https://github.com/adamborowski/explorejs here is demonstration of integration with Dygrahps, please see https://youtu.be/b_VRyhak5Xg It is not yet documented but there are many examples of many charts libraries integration Cheers, Adam