Making widgets talk to each other!

[ramnathv]

I have been experimenting with the ability to use widgets as components and then allow them to communicate with each other.

Here are some examples

1. Life Expectancy Choropleth + Line
2. Datamaps + Chroniton
3. Datamaps + Morris + Sortable

The first example uses shiny, while the second does not.

In the case of Shiny, I am using Shiny.onInputChange to trigger changes and communicate back to the server from the client on mouseover. In the second example, I am directly manipulating the datamap from the callback code of the chroniton widget.

The more I think about this, I believe that something like a pubsub design pattern will be helpful here. htmlwidgets can provide basic pubsub infrastructure and widget authors can provide callbacks that allow users to broadcast data on certain events. In a similar vein, widget authors can also expose callbacks that allow a widget to listen to certain events and trigger a change.

By following some conventions regarding how these events are named, it would be possible for widgets to talk to each other using pubsub. Currently, this idea is still very hazy in my mind, but I wanted to put it out here so that it can trigger a broader discussion.

In the shiny case, things are easier, since Shiny.onInputChange provides a simple pubsub mechanism. I am thinking of a more general mechanism that will work with and without Shiny.

@jcheng5 I would really appreciate any thoughts you have on this, given your experience working with javascript libraries. @jjallaire @timelyportfolio @yihui Any thoughts/comments on this idea would be welcome, as I think it would be a very powerful way forward for htmlwidgets to create a componentized architecture.

[timelyportfolio]

I agree pubsub would most likely be the best pattern. While crossfilter is nice in dc.js I think it will prove too limited for multiple htmlwidgets and difficult to teach to users. Ideally, the same event information will get passed to Shiny.onInputChange in shiny mode as our pubsub engine when not in Shiny mode, or we could allow the pubsub to notify shiny.onInputChange. I have been monitoring the work on vega closely and experimenting with various pubsub libraries. I don't have a favorite.

[abresler]

This looks amazing I can't wait

---

Alex Bresler abresler@asbcllc.com ​www.asbcllc.com​ 917-455-0239​ (cell)​ On Mar 5, 2015 8:10 PM, "Ramnath Vaidyanathan" notifications@github.com wrote:

I have been experimenting with the ability to use widgets as components and then allow them to communicate with each other.

Here are some examples

1. Life Expectancy Choropleth + Line https://www.youtube.com/watch?v=fbmSHQrrJrI
2. Datamaps + Chroniton https://t.co/9KzVYkbErf

The first example uses shiny, while the second does not.

In the case of Shiny, I am using Shiny.onInputChange to trigger changes and communicate back to the server from the client on mouseover. In the second example, I am directly manipulating the datamap from the callback code of the chroniton widget.

The more I think about this, I believe that something like a pubsub design pattern will be helpful here. htmlwidgets can provide basic pubsub infrastructure and widget authors can provide callbacks that allow users to broadcast data on certain events. In a similar vein, widget authors can also expose callbacks that allow a widget to listen to certain events and trigger a change.

By following some conventions regarding how these events are named, it would be possible for widgets to talk to each other using pubsub. Currently, this idea is still very hazy in my mind, but I wanted to put it out here so that it can trigger a broader discussion.

In the shiny case, things are easier, since Shiny.onInputChange provides a simple pubsub mechanism. I am thinking of a more general mechanism that will work with and without Shiny.

@jcheng5 https://github.com/jcheng5 I would really appreciate any thoughts you have on this, given your experience working with javascript libraries. @jjallaire https://github.com/jjallaire @timelyportfolio https://github.com/timelyportfolio @yihui https://github.com/yihui Any thoughts/comments on this idea would be welcome, as I think it would be a very powerful way forward for htmlwidgets to create a componentized architecture.

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/86.

[ramnathv]

Here is another example using the pubsubz architecture.

http://bl.ocks.org/ramnathv/raw/51f61a43f81910b868a5/

The idea is rather simple. The morris and dimple widgets listens to an event named elid_group and updates the chart accordingly. The datamaps widget broadcasts the state being hovered on to both these widgets to effect the change. It is very similar to what Shiny.onInputChange does. The question is can we write these events in such a way that both static and shiny contexts can take advantage of it.

[happyshows]

@ramnathv is there any updates on how htmlwidgets will communicate with each other in future? Also, could you share the code for the examples?

[ramnathv]

@happyshows this is a hard problem to solve in general. So it will take some time before we have more clarity on this. As for code, these are all widgets under development. So once I push them to github, I will also release the examples.

[happyshows]

@ramnathv thanks. In future will these code be under htmlwidgets or independent repo? I'm just trying to find a way to trace the progress.

[timelyportfolio]

Just circling back on this. Any thoughts on how to implement this?

[ramnathv]

The more I think of it, pubsub seems to be the answer. I will write up whatever I have discovered so far so that we can push this forward.

[jcheng5]

FWIW, I'm planning to look at this problem in earnest in October, probably in collaboration with @jjallaire and @hadley. I have a hard time imagining how the answer would not involve some kind of pubsub/event bus, but the semantics will need to be chosen very carefully.

[ramnathv]

@jcheng5 I would love to be a part of this discussion if possible.

[jcheng5]

@ramnathv Yeah, for sure.

[happyshows]

@ramnathv I'm also interested in clippyr (visual help guide) , is it on github under different name?

[ramnathv]

@happyshows I will put it up on github this weekend.

[happyshows]

@ramnathv , is clippyr on github now?

[timelyportfolio]

@jcheng5 & @ramnathv , it's October :) and I'm ready when you are for this discussion.

[jcheng5]

Just want to let you guys know that I am actively working on this. @jjallaire, @hadley, @wch, and I were all together in person last week and discussed this at length. I'll soon have a strawman writeup and prototype to serve as a starting point for further discussion.

[ramnathv]

Great! Looking forward to this @jcheng5 !

[timelyportfolio]

very happy to hear. let me know if I can help.

[jcheng5]

Sorry for the delay. I had to reboot this effort twice, I didn't realize how big the solution space was when I first started thinking about this and it took a while to get to an approach that was both practical and flexible.

I'm thinking about things in terms of a "near-term plan" that is actionable today and involves code we definitely can all agree needs to be written, but is modestly scaled in terms of ambition. Then we wait to see what kind of emergent behavior comes out of those improvements, and what opportunities we have to improve htmlwidgets to help out. Though below I also propose one possible direction that might be interesting to explore.

Near-term plan

The approach I'm advocating has two "prongs" today that address the needs of two different types of user. One is the R user who doesn't know JavaScript, and the other is the intermediate-to-expert JavaScript programmer.

For R users who don't know JavaScript:

We want to introduce a mechanism that will allow them to do things like linked brushing, linked identify, and dc.js/crossfilter-style filtering, without any effort on the users' part; just rendering three widgets that share a (new) "group" attribute value should link them together. Enabling this will take a significant amount of effort on widget authors' parts though, and we will have to provide both education and encouragement. The payoff is pretty huge if we can get this right.

That's the ultimate goal. The first step we want to take in this direction is to introduce a layer that different widgets in a page can use to speak to each other. I've built a prototype of such a layer in a tiny package I'm calling crosstalk. There's a NOTES.md document that outlines the capabilities, and the repo is here: https://github.com/rstudio/crosstalk

(BTW, for posterity, the link to the intro document is pinned to a specific commit; so if you're looking for the most up-to-date version of that file, be sure to go to the master branch first.)

I threw together a minimal d3 scatter plot package to explore these ideas without getting distracted by the complexity of most real-world widgets. The repo is at https://github.com/jcheng5/d3scatter and you can see an example of it in action with crosstalk here: http://rpubs.com/jcheng/crosstalk-demo

Crosstalk is designed not just for widget-to-widget communication, but also widget-to-Shiny and Shiny-to-widget communication as well. Try the examples on the d3scatter README. I think it'd also be very easy to extent Crosstalk to support other viz frameworks like RCloud, Bokeh.js, and Plotly.

I would love to get feedback on this approach over the next month. The high-level selection API mentioned in the NOTES.md still needs to be completed; I'm hoping to finish that next week.

For JS programmers:

The above approach is only going to be helpful for some common, standardized behaviors. It's not that helpful for totally ad-hoc interactions between widgets. For programmers proficient in both R and JavaScript, it can be challenging to get htmlwidgets to work together even with custom JavaScript code, without resorting to some ugly hacks. Primarily this is because we don't provide an easy way for htmlwidgets to have arbitrary methods on their objects; and even if we did, we don't have an easy way for regular JavaScript code to get ahold of widget objects. I have addressed the first part of that in this PR: https://github.com/ramnathv/htmlwidgets/pull/171

If we agree on that approach the second part of it (making it easy to get ahold of widget objects) will be straightforward. If these changes are noncontroversial they could be on the master branch before the end of next week.

Future plans

So non-JS-wielding R users will be able to do basic interactivity, and JS-savvy R users will be able to do anything they want (by writing custom JS code). I suspect there may be a fertile ground in between, where we can allow non-JS R users to do some declarative stuff in R that is way more flexible than just basic interactivity, but still keeps them from having to learn JS. I'm thinking maybe something akin to the Cocoa Target-Action pattern. This is pretty speculative though, and depends on the adoption of Crosstalk in the first place. So we'll wait to see what the reaction is to the first two prongs before we move forward with this more ambitious idea.

---

Feedback appreciated. If this isn't making sense to people I'm happy to vchat, or make a brief screencast or something, next week after the Thanksgiving holiday.

[jjallaire]

@hafen and @cpsievert this is a general purpose mechanism for widget-to-widget and shiny-to-widget communication that we are working on now. We'd love to make sure that there are no major impedance problems with rbokeh and plotly as getting those packages to work with this scheme would be a huge win. Let us know your thoughts.

[jjallaire]

@hafen and @cpsievert , The overall design is here:

https://github.com/rstudio/crosstalk/blob/master/NOTES.md

And here is a demo of what the R code currently looks like (with and without Shiny):

https://github.com/jcheng5/d3scatter/blob/master/README.md

[timelyportfolio]

@jcheng5, @ramnathv will group be the universal argument across all htmlwidgets to express which htmlwidgets will listen on the same "channel"? I only ask because I know that in rCharts we used group/groups as an argument similar to lattice. If we plan for group to be the argument, then we will need to change our rCharts2 API. Thanks.

[jcheng5]

Yeah leaflet has a similar problem with the word "group". We can change it.

[timelyportfolio]

What do other pubsub implementations use? Would channel be an option?

[timelyportfolio]

Another question out of ignorance, would this work with proxy mechanisms such as those in leaflet and I think soon in visNetwork? I would think yes, but thought I would bring it up just in case.

[jcheng5]

It really depends on the implementation to do it correctly, but I would imagine yes.

[jcheng5]

The "For JS programmers" section from https://github.com/ramnathv/htmlwidgets/issues/86#issuecomment-159863989 is now feature complete in #172.

[cpsievert]

Great stuff @jcheng5! So excited to work on this. I have some work in progress here and have some basic examples here.

Make sure you install the right branch before running examples:

devtools::install_github("ropensci/plotly@feature/events")

A few things/questions:

(1) For brush events, plotlyjs natively supports one mode: zoom. I have mentioned to the plotlyjs engineers (@etpinard, @alexcjohnson, @chriddyp, @cldougl, et. al.) that adding support for more modes (e.g., select box a la bokeh) would be a huge help, and there is intention on working on this. (2) group has a different meaning in plot_ly(), so I used set instead (this can change, of course). (3) Is there an elegant way to make an inline-block of htmlwidgets via htmltools?

[cpsievert]

I'm also having a bit of trouble figuring out how to pass selections from plotly to shiny without the use of an input brush. Is this what crosstalk::ClientValue is for?

[jcheng5]

Yes, ClientValue is the low-level abstraction for getting crosstalk values into R. I'm also working on a SharedData abstraction that is specifically for data frames with selection. I should be able to send you some sample code tonight or tomorrow. If that doesn't make it natural for you I'd love to vchat with you this week and understand a little more about the plotly JS API.

[cpsievert]

That'd be great! I imagine it works something like this? For some reason I get NULL even after clicking on points

[jcheng5]

@cpsievert It looks to me like:

1. When building the widget payload p in the R code, you're not actually including the set value, around this area: https://github.com/ropensci/plotly/pull/312/files#diff-eb044f253f9369d7d0711127e6035fbbR100
2. In your renderValue, I set a breakpoint on the ctgrp.var("tdb").set(keys); line, and it never seems to be hit.
3. In your server.R, crosstalk::ClientValue$new("tdb") should be crosstalk::ClientValue$new("tdb", group = "A").

If you load your app and from the JS console you run crosstalk.group("A").var("tdb").set("foo") you should see the output update.

[cpsievert]

Thanks @jcheng5! I fixed (1) and (3), and I think (2) was happening because I wasn't extracting the key properly for that example. It seems like it could be useful to expose more than just the key, so in https://github.com/ropensci/plotly/commit/d0b61180cf85e78566c809fc95942bd72e6cd8a1, I'm just trying to set/get all the point data, which seems to work client-side, but I'm getting Uncaught TypeError: Converting circular structure to JSON from shiny.min.js

Any ideas?

PS: I changed the naming a bit, but I revised the gist to reflect that

[cpsievert]

Actually, I think this is a plotlyjs issue. If I try to call JSON.stringify(data.points) from here it gives me the same error. Any ideas @etpinard / @alexcjohnson / @chriddyp / @jackparmer?

[jackparmer]

++ @mdtusz ^^^

[etpinard]

@cpsievert data.points[i].xaxis and data.points[i].yaxis are indeed circular structures.

They refer to the points' corresponding cartesian axis objects which you shouldn't need for your use case.

I'd recommend doing something like:

data.points.forEach(function(pt) {
  delete pt.xaxis;
  delete pt.yaxis;
});
JSON.stringify(data);

[alexcjohnson]

@cpsievert @etpinard even better, pull out just the pieces you really want from data.points. Each point has the COMPLETE trace and fullData (after supplyDefaults) trace, the latter of which contains references to code in _module... wayyyy more than you want. I'd prefer:

JSON.stringify(data.points.map(function(pt) {
    return {
        curveNumber: pt.curveNumber,
        pointNumber: pt.pointNumber,
        x: pt.x,
        y: pt.y,
        key: pt.data.myKey[pt.pointNumber] // if you need something else as the key
    };
});

[cpsievert]

Thanks @etpinard @alexcjohnson. I agree that we should send as little data as possible, so I bootstrapped on @alexcjohnson suggestion in https://github.com/ropensci/plotly/commit/d180812556ad03f1255913eadc2744cfc601f4bf

Here is a demo of handling plotly click events in shiny via crosstalk (source).

[jcheng5]

Nice work @cpsievert. The demo app code looks really clean.

You could simplify further by getting rid of the rv reactive expression; you can declare cv as a variable directly inside the server function, and use cv$get() from any reactive expression, observer, or output you wish. In fact, even if you decide to keep the rv reactive for future purposes, you should still hoist the cv <- ClientValue::new(...) declaration at the top level of the server function, and just put cv$get() inside the body of rv.

[jonathancallahan]

Just wanted to say that I'm very excited to see this work progress. I see mention of widget-widget and widget-shiny interactions but I didn't see any mention of "widget-R".

Will this work allow us to create widgets that interact with the RStudio environment/console?

I'm imagining an interactive widget that allows people to visually select some subset from a dataframe and that selection or that subset will somehow be communicated back to the RStudio environment. Alternatively, a widget be could used as a point-and-click interface to create an R expression that gets loaded into the console.

We are already creating a package that harnesses the dygraph and leaflet widgets to basically customize RStudio into a statistical analysis tool for non-programmer, Forest Service Personnel. It sure would be awesome if there were a mechanism by which scientists could click on a monitoring site in the leaflet widget and have their selection immediately available in the current R environment.

[jcheng5]

@jonathancallahan Yes, but you don't even need htmlwidgets for that--anything you can do in Shiny can be harnessed by the current R environment.

Try this (requires devtools::install_github("rstudio/shinygadgets"); and looks best in RStudio, although it'll work with any R):

library(shiny)
library(shinygadgets)
library(leaflet)

ui <- fillPage(
  leafletOutput("map", height = "100%")
)

server <- function(input, output, session) {
  output$map <- renderLeaflet({
    leaflet(quakes) %>% addTiles() %>% addCircleMarkers(layerId = as.character(1:nrow(quakes)))
  })

  observeEvent(input$map_marker_click, {
    stopApp(quakes[as.numeric(input$map_marker_click$id),])
  })
}

runGadget(ui, server)

[hafen]

Sorry I'm late to the game - been doing an insane amount of travel and catch-up when not traveling. This looks awesome and I'm really excited to dig into it. I'll be more involved at the beginning of the year.

[timelyportfolio]

@ramnathv, any progress on a new scaffold template? I'm almost thinking of working on this as my htmlwidget of the week. Let me know if I can help.

[timelyportfolio]

Or, I could convert one with a writeup to publicize the new direction for the year. Any particular htmlwidgets that provide an easy use case? Would knob be appropriate since it was a test for the first implementation?

[ramnathv]

@timelyportfolio I already have a local branch with the new scaffold template. I am on vacation currently, but will push it out over the weekend. I will be including an example as well.

[timelyportfolio]

Now that my widget per week commitment is over, I can play with crosstalk :) Here is d3scatter with parcoords http://bl.ocks.org/timelyportfolio/4c5718a7efe5c0abd363. Currently, I only have send not receive for parcoords. Thanks so much @jcheng5 for spearheading this.

[happyshows]

@timelyportfolio great achievement for 2015 :D Could you setup a voting app with multi choices for each widget and improve the highly rated ones later on?

[timelyportfolio]

@happyshows, thanks for all your usage/feedback on htmlwidgets. I had hoped to get a feel for popularity by Github stars, but unfortunately getting stars on Github for R projects seems nearly impossible. Unless otherwise motivated by feedback, ideas, use cases, I'll probably just pick my favorites.

[hafen]

@timelyportfolio congrats on getting through the year of htmlwidgets! I'd love to get all your missing widgets onto the htmlwidgets gallery. If we made the site more prominent and added a gallery, etc., then maybe it would become a place for things like github stars to become more effective.

[happyshows]

@timelyportfolio Maybe put a survey monkey link on your site to collect preference & feedback data instead? just my 2 cents.