Data manipulation in data views

[spatialillusions]

Hi, This weekend I was working with some code to create histograms from a data set, basically I created a function where I sent in a timeline data set, a step and a scale, and it returned a new dataset with my data grouped and counted so that I just could add it to a Graph2d. (There is a demo at spatialillusions.com/Demos/histogram ) This got me thinking that it would be nice to be able to do this more like a data view in vis, because it is a kind of view of my data, and in a similar way it would be nice to be able to view the sliding mean of a dataset as a data view. And then users could add all kinds of view functions to analyze and visualize their data. I don't think this is possible with the current data filters in data views, but please correct me if I'm wrong. Do you think this is a good idea, or should users add this kind of functionality in custom code?

[AlexDM0]

Hi Spatial,

The dataviews are not made for aggregating data as far as I know. We are working on new clustering methods for timeline and the network and we might bring them to the graph2d as well.

In regards to your current implementation with a zoomable graph I have a few comments. From a little profiling it seems that your clustering code takes about half a second while drawing the data only takes 25 milliseconds. I think you can greatly speed up your clustering algorithm with a little preprocessing and dataformats.

Just a pointer but you could keep track of an easily iterable array (to avoid the forEach which is slow) and preprocess a few date values so you dont need moment (which is VERY slow). I think you can make this near real-time with some optimizations.

Regards,

Alex

[spatialillusions]

Hi Alex,

I know my code could be optimized, it was just a quick coding done during the weekend to see if I could get it working. I did see that I could make the code faster, but since it kind of worked for my datasets, I decided to just be lazy. ;-)

I also know that you are looking into new clustering methods, and they will propbably solve the histogram case, but cases where I want to show a moving average for a data set, or other statistical functions won't be solved in that way. But maybe it's easier to build something else than a dataview that calculates the statistics for you dataset, and then just trigger an update of that when your dataset is updated.

/Måns

[josdejong]

Can be interesting to extend the DataView with capabilities to aggregate data. Right now it supports map and filter functionality, we could add this with a reduce step or something like that.

Besides the DataSet and DataView, we have been thinking about a DataTransform. This isn't implemented yet but is an interesting concept if you ask me. Basically a DataTransform should define a relation between original data and transformed data in two directions. So it would not only allow you to create a transformed view of your data like DataView does right now, but also allows to add/update/remove data and knows how to propagate that back to the underlying DataSet.

[spatialillusions]

DataTransform sound like a nice concept! Hopefully you will get time to implement something like that.

BTW, I updated my code so that it's a whole lot faster, storing moment objects directly in the dataset saved a lot of time. I will keep updating the code to make it more general and adding things like max,min, mean, median, sum, count for each cluster. One thing that I have realised during the development, is that it would be really nice to be able to select a bar in the bar chart and get the data from your dataset, like you are able to select items in the timeline.

Please let me know if you think that I can be of any help with implementing clustering in timeline/graph2d.

/Måns

[josdejong]

Sounds good. If we need help with the clustering we will let you know, thanks. Alex has just reworked the clustering for Network, it looks promising and we can apply something similar to Timeline.