Simpler Widgets

[ramnathv]

I wanted to initiate a discussion on a feature that will allow users to create widgets without an elaborate package infrastructure. I believe, this will be useful at the very least to lower the barriers to entry for development. I have ongoing work on this in this feature branch

https://github.com/ramnathv/htmlwidgets/tree/feature/simpler-widgets

The basic idea is to allow a widget to be specified as a self-contained folder with a special structure.

|- widget
  |- hello.R
  |- htmlwidgets
    |- hello.js
    |- hello.yaml

The biggest (and probably only) advantage we get from requiring widgets to be stored in a package is the ability to locate the dependencies consistently. However, I believe this can also be achieved by storing widgets in a consistent location, say ~/.htmlwidgets, which is the technique used by command line tools like node and homebrew.

I see significant advantages to supporting this feature:

1. Developing widgets is as simple as writing an R + JS file. No package development tools needed. This can lower the barriers and probably also motivate JS developers to provide simple R scripts that will allow consumption of their libraries.
2. Widgets can be installed from gists and zip folders. This is particularly useful for small utility widgets that do not justify having an entire package installed.

I know that @jeroenooms brought this up before and there was some discussion around why packages are better. I agree that packages should be the primary and preferred mechanism to develop widgets. However, I also believe that providing a simpler folder-based mechanism will foster more experimentation and lower barriers to entry.

Let me know what you guys think. @jjallaire @jcheng5 @timelyportfolio @yihui @hadley @hrbrmstr @hafen

[jcheng5]

I think there's merit to this idea, especially for experimentation and proof of concepts. But there are at least a few other advantages to using a package that are worth enumerating.

1. Metadata (version [especially], author, etc.)
2. Declaration of package dependencies, in a format that R/devtools can use to fulfill those dependencies.
3. Namespacing--without this you will need to mess with the user's search path (or use :: everywhere I guess)
4. Ability to document functions in a way that ties into R's help system
5. Control over which functions are visible to the user (exported vs. unexported)
6. Consistent place to put tests

(Also, I don't think gists can contain subdirectories, so hello.R/hello.js/hello.yaml might all have to be peers.)

I don't think it'd be crazy to say that you start with a simpler layout like you're proposing, but the minute something becomes useful enough that it wants to be in an actual GitHub repo, it's time to make a package.

[jjallaire]

I don't think you'd ever want to distribute a widget that had no help and no namespacing, so this is really for experimental development scenarios. However with scaffoldWidget + devtools/RStudio it's trivial to get a simple widget up and going and do quick iterative development on it.

I also think it's important to note that most widgets will define multiple interrelated R functions and have substantial external dependencies (JS files, etc.), both things that packages excel at.

That said, I don't see a reason to oppose doing this unless it requires refactoring that breaks existing code (a distinct possibility depending on what's required to make this possible).

On Thu, Jan 7, 2016 at 3:34 PM, Joe Cheng notifications@github.com wrote:

I think there's merit to this idea, especially for experimentation and proof of concepts. But there are at least a few other advantages to using a package that are worth enumerating.

1. Metadata (version [especially], author, etc.)
2. Declaration of package dependencies, in a format that R/devtools can use to fulfill those dependencies.
3. Namespacing--without this you will need to mess with the user's search path (or use :: everywhere I guess)
4. Ability to document functions in a way that ties into R's help system
5. Control over which functions are visible to the user (exported vs. unexported)
6. Consistent place to put tests

(Also, I don't think gists can contain subdirectories, so hello.R/hello.js/hello.yaml might all have to be peers.)

I don't think it'd be crazy to say that you start with a simpler layout like you're proposing, but the minute something becomes useful enough that it wants to be in an actual GitHub repo, it's time to make a package.

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/176#issuecomment-169797493 .

[ramnathv]

Thanks for all the feedback @jcheng5 and @jjallaire. I completely agree that packages are and will be the preferred route to distributing widgets.

My key motivation for this work was to facilitate easier experimentation and more rapid iteration of the widget development process, since I did not want to keep going through the edit-build-test cycle, which is required with packages (although it is really simple with devtools/rstudio combo).

It touches very little code, so I don't anticipate any breakages to existing code. I will do some more tests to confirm the same. I will also add additional functionality that will allow a user to develop a simple widget and copy it over to a package. This will bridge the gap between experimentation and distribution.

This PR is still a few cycles from being complete. I will ping the group again once I have had the chance to polish it further.

[timelyportfolio]

I would love to see this even though I agree packages are by far the best route. After > 52 and very great RStudio tools, packages are easy for me now :). Not sure if worth bringing back up #1. I'll think more deeply through this tonight and try to draft a more intelligent response.

[jjallaire]

Being able to just edit JS and refresh the browser would be a nice workflow improvement. If it's not too disruptive to the existing codepaths I'm all for it.

On Thu, Jan 7, 2016 at 4:30 PM, Ramnath Vaidyanathan < notifications@github.com> wrote:

Thanks for all the feedback @jcheng5 https://github.com/jcheng5 and @jjallaire https://github.com/jjallaire. I completely agree that packages are and will be the preferred route to distributing widgets.

My key motivation for this work was to facilitate easier experimentation and more rapid iteration of the widget development process, since I did not want to keep going through the edit-build-test cycle, which is required with packages (although it is really simple with devtools/rstudio combo).

It touches very little code, so I don't anticipate any breakages to existing code. I will do some more tests to confirm the same. I will also add additional functionality that will allow a user to develop a simple widget and copy it over to a package. This will bridge the gap between experimentation and distribution.

This PR is still a few cycles from being complete. I will ping the group again once I have had the chance to polish it further.

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/176#issuecomment-169810345 .

[ramnathv]

Yeah @jjallaire That is another benefit I was going after. In fact, I think with some more tooling, one can set up a live-reload sort of environment, where changes to the R or JS files will automatically reload the preview. Thanks for the idea.

[jjallaire]

That would be spectacular!

On Thu, Jan 7, 2016 at 5:00 PM, Ramnath Vaidyanathan < notifications@github.com> wrote:

Yeah @jjallaire https://github.com/jjallaire That is another benefit I was going after. In fact, I think with some more tooling, one can set up a live-reload sort of environment, where changes to the R or JS files will automatically reload the preview. Thanks for the idea.

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/176#issuecomment-169819810 .

[ramnathv]

I would need to pick @yihui 's brain on this as I remember him setting up some cool reloading stuff with servr::jekyll(), where saving updates to an Rmd file automatically triggered rebuilding the html and update the preview.

[jcheng5]

Hot reloading the code would be cool but I would prefer if it was orthogonal to the packaging format, i.e. we should make it work for package based widgets too if possible.

(When I want to work in this way currently, I save a sample widget HTML page and put it in the htmlwidgets directory, and change the dependency URLs to be relative. It works but is clunky obviously.)

[ramnathv]

That makes sense @jcheng5. Once I am able to get the hot reloading working for the folder based widgets, I think it should not be too hard to extrapolate it for package based widgets, since the main difference would be the additional build step for the package.

[jjallaire]

I wonder if we could do something like enable a global option which would cause widget dependencies to be loaded from the package source directory (which we could find via sourcerefs). @ramnathv if we did this then for JS only changes you wouldn't even need to rebuild the package.

Come to think of it devtools::load_all might actually override system.file so all that might be needed is to detect the changes then execute devtools::load_all.

On Thu, Jan 7, 2016 at 5:06 PM, Joe Cheng notifications@github.com wrote:

Hot reloading the code would be cool but I would prefer if it was orthogonal to the packaging format, i.e. we should make it work for package based widgets too if possible.

(When I want to work in this way currently, I save a sample widget HTML page and put it in the htmlwidgets directory, and change the dependency URLs to be relative. It works but is clunky obviously.)

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/176#issuecomment-169821388 .

[timelyportfolio]

Of course, if we go this route, gistr from @sckott will be very helpful to quickly distribute.

[timelyportfolio]

... and I thought 2015 was fun, just wait for 2016

[ramnathv]

That makes sense @jjallaire. Enabling that will be awesome.

[ramnathv]

I was able to figured out a trivial implementation for the folder based widgets using servr::make. Is a make dependency acceptable for this?

[jjallaire]

I think it's development time only it's a fine dependency. Note that you can probably locate make on windows by sniffing the registry for the location of Rtools.

On Thu, Jan 7, 2016 at 5:30 PM, Ramnath Vaidyanathan < notifications@github.com> wrote:

I was able to figured out a trivial implementation for the folder based widgets using servr::make. Is a make dependency acceptable for this?

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/176#issuecomment-169827639 .

[jcheng5]

The approach I was imagining wouldn't involve make, would work for both folder-based or package-based, and for the latter wouldn't involve rebuilding the package. The only dependency would be httpuv (and could be a Suggests). Are you around to vchat about this? Or maybe I can put together a gist to demonstrate.

[ramnathv]

That's neat @jcheng5. I have a meeting till around 3:30, but can vchat after that. Will you be around in half hour?

[jcheng5]

Sure, ttyt.

[jcheng5]

Per our chat: https://gist.github.com/jcheng5/be8bc7256e463e977596

The rootUrlHandler param would render the widget, fixing up the widget dependencies to point to relative URLs (renderDocument takes a processDeps parameter that could be used for this). The otherUrlHandler would resolve URLs (basically /favicon.ico plus the htmlDependency URLs you came up with during processDeps).

shouldReload would be a stateful closure that checks whether any of the files under {dir}/htmlwidgets/ (or whatever) has changed.

[hafen]

@jcheng5 this is awesome! A question would be whether live reloading functionality should be part of htmlwidgets or servr or its own package. It'd be great to be able to use outside of htmlwidets. For example, I'd like to be able to use it here: https://github.com/hafen/rmote.

[ramnathv]

@hafen I agree that this functionality would be useful beyond htmlwidgets. But the piece we are focusing on is highly tied to htmlwidgets and I think should be housed in the htmlwidgets package, as one of of its main roles would be to facilitate faster and more interactive development of widgets.

[yihui]

That implementation is very similar to what I did in servr.

[hafen]

Oops I didn't realize that already existed in servr.

[ramnathv]

Here is a proof-of-concept that combines the idea of folder based widgets and live-reloading using servr::make().

https://www.youtube.com/watch?v=E6Wl-bqRSAU

You can play with it by installing htmlwidgets from the feature/simpler-widgets branch.

I had to use make to ensure that the widget html is regenerated from the R source, whenever any of the dependencies change, but based on the earlier note by @jcheng5, i believe this can be rewritten to just make use of httpuv.

Let me know if you have any comments/feedback. I believe this feature would significantly lower barriers to widget authoring and development (even for experts like @timelyportfolio, @hafen).

[timelyportfolio]

I hope to have some time to test this out this afternoon. I will report back. Thanks so much @ramnathv for implementing this.

[timelyportfolio]

@ramnathv, besides the line notes, it worked perfectly. I tested it out on trianglify and had a live coding environment with a working widget in < 5 minutes. I like it very much. Now, I am wondering are there ways we can easily smooth the transition to a real package from this simple htmlwidget?

[jjallaire]

I think @jcheng5 alluded to the possibility that we could get the exact same pipeline working for in-package widgets. Joe, is that correct?

On Fri, Jan 29, 2016 at 4:39 PM timelyportfolio notifications@github.com wrote:

@ramnathv https://github.com/ramnathv, besides the line notes, it worked perfectly. I tested it out on trianglify and had a live coding environment with a working widget in < 5 minutes. I like it very much. Now, I am wondering are there ways we can easily smooth the transition to a real package from this simple htmlwidget?

— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/176#issuecomment-176980259 .

[ramnathv]

Yes @jjallaire. That is my understanding as well.

[timelyportfolio]

Also, I'd like to remember mason from @gaborcsardi and see how we might integrate with it.

[timelyportfolio]

Here is my awesome, incredible widget if anybody would like to see https://github.com/timelyportfolio/simple_widget_tests.

[ramnathv]

@timelyportfolio i have some thoughts on an adapter function that will transfer the widget over to a package. That way a user can mock up a simple widget in a folder and then seamlessly transfer it over to a package. With live-reloading enabled for widgets housed in a package, further tweaks to the widget can also be handled easily.

[gaborcsardi]

Just skimmed over the discussion above.

I am happy to write an htmlwidget template to https://github.com/metacran/mason if you just want to try it. (Although it works best in the console, unfortunately.)

[Personally I think that with some nice tool(s) to handle the package infrastructure having packages is not such a big burden. E.g. an RStudio addin with a simple form (with reasonable defaults) to create/update the package for a widget.]

[timelyportfolio]

@ramnathv, should we include the Shiny bits in the simple widgets?

[ramnathv]

@gaborcsardi I think everyone is in agreement that packages are the best way to ship widgets, and given the infrastructure put together by @rstudio, it is no longer a daunting task. My rationale for simple widgets was to avoid the edit-build-run cycle that is often involved while building a widget. Moreover, it was simpler to implement live-reloading with this structure. But ultimately, widgets will have to reside in a package, and it will be nice to think through how mason or a mason based addin can help make it easier and more flexible.

[ramnathv]

@timelyportfolio I would keep the shiny bits in, so that it becomes easier to transfer the widget over to a package. I have not even tested if the shiny bits work with simple widgets, but it should not be too hard doing that.

[timelyportfolio]

updates?