mdsumner
commented
7 years ago Thanks Miles, I'm a bit out of my depth in the visualization tech. but I believe that there are two main and potentially opposing goals here. One is meaningful interactive graphics and the other is a very generic flexible and efficient language of data representation. I think we must find a way to marry these two topics, and I do think a solution lies in this integration.
Every geometric feature in ggplot2 and in sp/sf is representable by path-based structures, these are planar turtle-head-down continuous traces between coordinates. They are generally treated as discrete closed rings or open connected lines, and any space-filling is provided by the planar "polypath" convention for winding or the even-odd rule. Ggplot2 does have a a convention for a sequence of line segments, treating the attribute on the first coordinate as "the value on the segment", and it doesn't otherwise have "true primitives" - in the WebGL or simplicial complex sense.
True primitives (triangles, line segments, tetrahedrons) allow models to incorporate both continuously varying and discrete properties in one structure, but this power tends to be watered down or removed in traditional implementations. For example, GIS gains great efficiency being bound to planar path forms, and it bakes in a single-level grouping to separate the geometry from the properties of the features. When GIS breaks out of this constraint it does it on the fly, or in ways that are specific to implementations (there's no standard for breaking-out and returning).
Rgl in comparison is (mostly) a simplicial complex form, with a (transpose) matrix of homogeneous coordinates indexed by a (transpose) matrix of the primitives. It's confusing though, because the two matrices are only seen explicitly in triangulation and quadrilateral forms (quads are not "proper" primitives, but they are used by GL as a convenience - IIUC). The line segment "meshes" in rgl are much simpler, and tend to be defined implicitly without any formal structures. These forms in rgl can have aesthetics applied, and generally discrete properties are those that are constant among a primitive's vertices, continuous ones can vary - the continuous mode includes the geometric representation of the primitive of course, as well as colours, shading properties, width and so on.
The indexed matrices in rgl are well represented in data frame form by relations between tables, and in this form there is a lot more scope for higher level groupings and structure. When it's done on data frames there's no limit in what properties can be stored, but we leave behind the simple convenience of a single data frame, but gain the generality and back-endability of database and memory management tools. I think there's a lot to be gained at the basic data-structure handling side, and what I've learnt from dplyr and tidygraph in particular, as well as sf suggests there's a lot of room for improvement at the R level, at least for learning what is possible.
Having more general and low level hooks into graphics engines like WebGL would go a long way I think, there are "scenes" that I can construct that are really valuable but are hard for me to share given the multi-step preparation or merging of different data, and the device requirements for rgl. I've found the WebGL write from rgl can only handle smallish content, and I don't know why that is, but also the navigation mode by default is "pivot on centre of data" which is no good for immersive exploration. Rgl is also very good at texture mapping, and the indexing is pretty straightforward to do, but it has to go through a PNG file and so is less flexible than R is usually for that kind of data.
R is already immensely capable of traversing and analysing these rich structures, and topological manipulation and analysis and physical modelling is a natural fit as well as a rich source of visualization.
MilesMcBain
dicook
DataStrategist
cjyetman
Okay I'll lead off by summarising some of things that have been said on Twitter, our email thread, and bobbing around in my own brain:
Get the basics right
From the conversation we've had so far, it sounds like there is consensus about not straying too far from the
ggplot2grammar of graphics interface for traditional 2D statistical plotting. As @hrbrmstr said, a lot of folks already struggle making useful visualisations for their analysis and sometimes interactivity is an unwelcome distraction. From this base we would like to build out well known interactivity mechanisms like zoom, brush, hover, and selection.Be Reactive?
Looking at this from the development UX perspective, I am really excited by the ideas Mike Bostock put forward in his D3.express reveal, and how we might incorporate reactive bindings to enable users to more rapidly learn the grammar of graphics. This is one way we could making building the appropriate vis easier.
Be Modular, Adaptable, Extensible
It was mentioned that previous efforts have fallen down by being closely coupled to graphics back-end that was not portable. Web technologies as the solution to this have been implicit in the conversation. There are multiple viable back-ends and it will be worth investigating whether a flexible decoupled architecture can give us the best of a few worlds (first suggested by @thomasp85). WebGl has been mentioned as worthy of interest. I firmly agree with @thomasp85 in that an extension mechanism needs to be baked in from the start.
Handle Streams Natively
My personal views, that relate to the source of the Tweet-storm, are that whatever comes next needs to handle visualisations of streaming data. This is useful for raw data vis and extremely important for diagnostics and predictions of sequential models. This requirement means the rendering framework needs to be FAST (Hello WebGl) and have some inherent concept of plots that are update-able in a more fundamental way than layering.
Lead the Way to What's Next
Can we create an interface that allows a reasonably seamless transition between 2D, 3D, AR, or VR data visualisation? There's a lot to bite off here, but extensions into this space would be a huge draw card for R.