Open dmurdoch opened 9 years ago
@dmurdoch Great to know that you want to make an rgl
widget. The approach you have indicated is broadly correct. To make things concrete, here are some suggestions.
writeWebGL
separating the javascript and the R code. The js code essentially would be a large js function that accepts a json object as its argument. You can put this inside the renderValue
method in the htmlwidgets/rgl.js
folder. You can use htmlwidgets::scaffoldWidget('rgl')
to set up a scaffold of all the components required for this widget. I would be happy to help with this widget. So let me know if you have any questions or run into any sort of troubles.
The one challenge I see here is that writeWebGL
seems to be writing directly to script
tags. For example, I pulled this off one of the examples.
<!-- ****** spheres object 6 ****** -->
<script id="vshader6" type="x-shader/x-vertex">
attribute vec3 aPos;
attribute vec4 aCol;
uniform mat4 mvMatrix;
uniform mat4 prMatrix;
varying vec4 vCol;
varying vec4 vPosition;
attribute vec3 aNorm;
uniform mat4 normMatrix;
varying vec3 vNormal;
void main(void) {
vPosition = mvMatrix * vec4(aPos, 1.);
gl_Position = prMatrix * vPosition;
vCol = aCol;
vNormal = normalize((normMatrix * vec4(aNorm, 1.)).xyz);
}
</script>
Injecting such a script
tag is not easy in htmlwidgets
. @jcheng5 any thoughts on what mechanisms will help this use case. One approach that came to my mind is to save the generated script
tag in a temporary javascript file and then use the attachment
mechanism to dynamically load it.
That's not really necessary, it's just the way the model code did it that I based writeWebGL on. I was planning to make it pure Javascript anyway: I just need to put that shader code in a string.
Good to know @dmurdoch. If you hit any roadblocks along the way of implementing this widget, do let us know.
I would be happy to help also but with @dmurdoch and @ramnathv not sure I can add much.
I have some questions about details and philosophy of the implementation of htmlwidgets. Some background first:
Currently, writeWebGL is monolithic, not distinguishing between initialization and rendering. So I need to decide where the dividing line goes.
My plan for the first pass is to mimic writeWebGL from the user point of view. The user will call a series of rgl functions creating a scene within R, then make a single function call to translate everything into WebGL for display in the widget.
Next I'll add functions to allow modifications to an existing scene, but not adding or deleting elements. These will be like the existing propertySetter() and related functions.
Finally I'll make it general, so the htmlwidget display in RStudio could be used as the main display while developing the scene.
So, what's initialization, and what's rendering? One idea is a very minimal initialization: just create an empty rglClass object. When renderValue is called, argument "x" will be the whole scene to be rendered. The other extreme is that the initialization does most of what writeWebGL does now, and the renderValue function hardly does anything.
Which makes sense?
And one other question: in RMarkdown documents, it makes sense that each new display can start with the previous one, and just make some modifications. How do I do this in htmlwidgets?
(Sorry for brevity, typing this from my phone while on vacation)
Initialize is called once, renderValue can be called many times (not in an RStudio or R Markdown context, but when used with Shiny). If new data should always cause your widget to be reset then essentially everything in your monolithic approach can go into renderValue and you can leave initialize empty. However if you want to, say, preserve the camera angle (or whatever it's called in WebGL) or you want to somehow diff the old and new data and only change the parts of the scene that are necessary, then you might need to do a different initialize/renderValue split.
For the second question, we do this by having the R representation of the widget be a list that you modify using accessor functions, with pass by value semantics. The dygraph and leaflet packages both use this approach.
Examples of how progressively building up a leaflet visualization are here: http://rstudio.github.io/leaflet/.
You'll notice two functionally equivalent syntaxes. Here's the traditional one passing the map as the first argument to various mutator functions:
m <- leaflet()
m #print empty map
m <- addTiles(m)
m # print map with tiles
m <- addMarkers(m, lng=174.768, lat=-36.852, popup="The birthplace of R")
m # print map with markers
The same transformations can also be done via the %>%
operator in a chain if you don't want to print the intermediate renderings:
leaflet() %>%
addTiles() %>%
addMarkers(lng=174.768, lat=-36.852, popup="The birthplace of R")
As Joe described there are different philosophies on whether to rebuild the world in renderValue
verses doing smart "diffing" of representations and providing smooth transformation. Part of this depends on the ability of the underlying library to handle the "diffs" and part depends on whether you expect the component to be used frequently in Shiny applications where smooth transformations based on varying inputs will be required. In dygraphs we rebuild the world every time largely based on the fact that dygraphs doesn't handle incremental updates very well. In leaflet things are handled incrementally because this is often very useful for exploring spatial data.
Thanks Joe and JJ. Since rgl scenes are so big, I think it makes sense to make the initialize method fairly heavy, and then each of the renderValue calls won't need to send much.
Now I have a more detailed question. (Remember that I don't really know what I'm doing in Javascript.)
I need to transfer 4 by 4 matrices from R into Javascript "CanvasMatrix4" objects. I have been doing it by generating code like
m = new CanvasMatrix4(); m.load([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]);
Can this happen automatically in JSON (e.g. with a custom serializer), or should I just pass an array, and convert it to a CanvasMatrix4 object with Javascript code like the above?
Can this happen automatically in JSON (e.g. with a custom serializer), or should I just pass an array, and convert it to a CanvasMatrix4 object with Javascript code like the above?
The latter. htmlwidgets will only give you generic JS scalars, arrays, and objects--it's entirely on the author to turn those into domain specific objects (if necessary).
One more thing to keep in mind: the distinction between initialize
and renderValue
only matters for use within Shiny (at the R console and in R Markdown both are called only once). You can still of course progressively display / build up a widget payload with successive calls that mutate the x
object but except for within Shiny every time you print the widget it's going to run both initialize
and renderValue
.
In the Shiny case however the widget is updated in place via the call to renderValue
. This gives you the opportunity to do intelligent diffing between current and previous states of x
and to only render the new options / data (and perhaps even do a nice visual transition from old to new state). However it should be emphasized that this sort of smart transition is going to be quite complex to code correctly. D3 provides a nice start for data transitions via it's enter/update/exit handlers however for many other libraries you'll need to do this from first principles. In some cases the library already supports this sort of dynamic update (in my case for dygraphs they advertised that they could do dynamic updates however in practice there were a bunch of exceptions to this so I ended up just re-rendering the world on every renderValue
).
I've looked at it in more detail, and now I have an approach in mind. The initialize call will just set up an empty scene. When I call renderValue, it will either send the whole scene, or just some updates, according to a flag in x. (Essentially the flag will say whether to clear the scene or not.)
I don't want to send the whole scene every time because that can be a lot of data, and I'm worried it will be too slow. My first attempt at a Shiny implementation (back in December, long before I'd heard of htmlwidgets) resent the whole scene, and it took a couple of seconds to make a change.
Not to complicate things too much further, but I think the direction you are going is very similar to what we ended up with for leaflet. In that case for the Shiny scenario we created a leafletProxy
function which provides a lightweight handle to an existing leaflet instance. All functions that work against leaflet
also work against leafletProxy
. See the docs here for usage examples: http://rstudio.github.io/leaflet/shiny.html
If the idea is to avoid the transfer of data from R to JavaScript then I think you'll need to do something like leafletProxy to avoid re-sending all of x
every time. If however the main issue is rendering time then the flag within x
is probably okay (that said, I think the proxy
pattern is a nice way for the user to express the desire for incremental updates within Shiny). We ended up doing the same thing for DataTables (see the code at the bottom of this example: https://yihui.shinyapps.io/DT-proxy).
@jcheng5 and @yihui Know more about the particulars of the proxy pattern so may want to add something here.
Thanks for the pointer. It's the transfer of data that's the issue; WebGL is quite fast at rendering once things are in the browser. I'll follow the leafletProxy model when I get to that part of the implementation.
I'm making some progress on this, but now my lack of Javascript knowledge is slowing me down. These are probably simple questions:
There are functions that allow you to do these operations. For (2), you can use the code used in d3.merge
.
d3.merge = function(arrays) {
var n = arrays.length,
m,
i = -1,
j = 0,
merged,
array;
while (++i < n) j += arrays[i].length;
merged = new Array(j);
while (--n >= 0) {
array = arrays[n];
m = array.length;
while (--m >= 0) {
merged[--j] = array[m];
}
}
return merged;
};
So d3.merge([[1, 2], [3, 4]])
will return [1, 2, 3, 4]
.
As for (3), the approach you are using is correct. You could wrap them up in to functions nrow
and ncol
so that the javascript code is R friendly.
I will try to add a solution for (1) later today.
For (1), I think you can use the concat
method.
x = [[1, 2], [3, 4]]
y = [[5, 6]]
x.concat(y)
console.log(x)
This will give
x = [[1, 2], [3, 4], [5, 6]]
Ok, deleted comment also
I'm making good progress on this: most types of object now display. I'm currently working on textures. I think I could get them to work similarly to the way rgl handles them (i.e. copy the image file into the same dir as the web page, load from there), but a better way would be to put the image directly into the .html source. Is there support for that in htmlwidgets?
Specifically I need to end up with a Javascript Image() object that contains an image from a file, but I don't want the Javascript to read the file, I want R to read the file.
We have an attachment API (not yet documented) that was put in for just this sort of scenario. Here's some informal documentation that @ramnathv provided to another developer:
https://github.com/ramnathv/htmlwidgets/issues/71
Does that look like what you need? @jcheng5 Anything else to add here regarding correct use of the API?
There is also a pending PR https://github.com/ramnathv/htmlwidgets/pull/83 that attempts to make this process a whole lot easier. I will review it this weekend to see if we can merge this soon, as it would make things much easier.
I think my question was more naive than you thought. I've found the knitr::image_uri function, and it does what I wanted.
For anyone coming along later: In R, do
x$uri <- knitr::image_uri(filename)
which puts a long string representation of the image into a field of the object to display. In Javascript, turn it into an image by
image = new Image();
image.src = x.uri;
That's all that's needed.
That's great, glad it was that simple!
On Sat, Aug 22, 2015 at 8:57 AM, dmurdoch notifications@github.com wrote:
I think my question was more naive than you thought. I've found the knitr::image_uri function, and it does what I wanted.
For anyone coming along later: In R, do
x$uri <- knitr::image_uri(filename)
which puts a long string representation of the image into a field of the object to display. In Javascript, turn it into an image by
image = new Image(); image.src = x.uri;
That's all that's needed.
— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/141#issuecomment-133700266 .
I've now got a mostly working version of the rglwidget, in a package of that name on R-forge. It requires an unreleased version of rgl, also on R-forge.
Long range plans are to backport many of the rglwidget changes to rgl, but that won't happen for a while.
In the meantime, I've got one problem I'd like to work on. I think if I just display my rglwidget in the RStudio console, the viewer should show the display, but it doesn't. Is there anything like the Firefox web console or debugger that I can use to see what's going wrong?
On Linux and WIndows, you should be able to right click on the widget page, and use Inspect
to debug the page. It is pretty much like Google Chrome's Developer Tools. WebGL may not work well in RStudio's viewer. At least it did not fully work when I was trying to display a 3D globe from the threejs package (https://github.com/bwlewis/rthreejs). What is your RStudio version number?
Thanks. I'm mainly using 0.99.467 on Mac OSX, so no Inspect there. I can run Windows XP in a VM, but current RStudio doesn't work well in XP. The Inspect page did work, but gave some spurious syntax errors. I can also run Ubuntu in a VM. I'm just installing 0.99.467 there.
I've basically got the rglwidget working in an Rmarkdown document now, so I wanted to get it going in Shiny. I've got two questions.
The last time I tried to do this (about 10 months ago, with rgl::writeWebGL writing the Javascript), I found the response time was too slow. I think this was because rgl scenes can really have a huge amount of data in them (up to megabytes), and the way I wrote it, this would all have to be downloaded again if the user made a change via Shiny. I want sliders that give near-instant responses.
Earlier this year I wrote various functions that took an existing scene and just modified a few of the values within it via pure Javascript; that was fast, since it didn't need to go back to the server.
The time lag may be less of a problem now with the rglwidget since the Javascript doesn't get re-sent with each change, but there's still a lot of data. I decided on the following approach to deal with it. If I want Shiny to control a scene without redrawing the whole thing, I would insert two output objects: one that draws the scene, and the other that receives commands from Shiny and modifies the first one. (The first is called "rglwidget", the second "rglcontroller".) The rglcontroller would be invisible.
First question: is this design a bad idea?
Assuming I go with the above design, the rglcontroller needs a way to refer to the rglwidget it is controlling. I had assumed I could use elementId for this, but I'm getting warnings:
Warning in func() :
Ignoring explicitly provided widget ID "thewidget"; Shiny doesn't use them
Second question: how should one Shiny output object refer to another one in Javascript?
Great! A couple of things that are relevant here:
renderValue
function, you can explicitly store your instance of the rglwidget
in the DOM by adding a line el.rglwidget = mywidgetinstance
. From now on, you will be able to access your widget instance by id.I think this may be where you need the "proxy" pattern implemented by Leaflet (it has a leafletProxy function than supports all the same operations as "leaflet" but references an existing instance). @jcheng5 or @yihui can describe this in more detail.
On Sun, Aug 30, 2015 at 1:26 PM, Ramnath Vaidyanathan < notifications@github.com> wrote:
Great! A couple of things that are relevant here:
1.
In your renderValue function, you can explicitly store your instance of the rglwidget in the DOM by adding a line el.rglwidget = mywidgetinstance. From now on, you will be able to access your widget instance by id. 2.
With Shiny, you can use several explicit mechanisms to control what is passed between client and server. See this https://ryouready.wordpress.com/2013/11/20/sending-data-from-client-to-server-and-back-using-shiny/ article for a nice description. I believe, this idea was used by @jcheng5 https://github.com/jcheng5 in the Leaflet package.
— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/141#issuecomment-136176345 .
Thanks, I've got it working now with the rglcontroller. Now I need to write some controllers so I can try it out.
I'll also take a look at the leaflet source to see if that way makes more sense.
The basic idea is that you add a custom handler on the JS side via Shiny.addCustomMessageHandler()
(leaflet example here), and use session$sendCustomMessage()
to send data from the R side to JS (here).
I'm quite happy with the way the rglwidget is going. I think the design is right; it's just catching up on all the details and documentation now.
This is the design:
There are two widgets: the rglwidget which displays the scene, and an invisible rglcontroller widget which receives messages from Shiny controls and uses them to change the rglwidget.
If the rglcontroller only needs the value of a slider or other input control, it doesn't bother using the reactive input$slider, it installs an onchange event handler on the slider. This speeds things up a lot, so the Shiny sliders can be used for smooth, real-time modification of the rgl scene. I love this! You guys have put a lot of work into the Shiny controls, and I just get to use it!
On the other hand, sometimes the change to the scene needs to be computed based on the inputs. That still works with reactive inputs.
This has been slow, but I think I'm getting a really big improvement to rgl out of it.
I have a question about the best approach to handle images to be used as textures in a WebGL scene.
The standard way to do this is in Javascript code: create an img, set its src to the image file, and set an onload method to install it as a texture. I'd like my HTML files to be self-contained, so I do the base64 conversion of the image and use a data URI as the image source.
However, the pandoc conversion from .md to .html is really slow. I think the long strings holding the base64 data are causing the slowdown. It goes much faster if I have <img src="file"\>
HTML code, and let pandoc do the base64 conversion when it makes the output self-contained.
So my question is: is there a way to insert HTML code like <img src="file"\>
when inserting a widget? The Advanced Topics vignette talks about custom widget HTML, but I don't think the function that generates the HTML gets to see the particular instance. The images will depend on what's in the rgl scene, to generate them I think I need to see the "x" argument to createWidget. Is there a way to retrieve it?
I think for this situation the recommended approach is usually:
1) Within your top level R function (the one that calls createWidget) capture whatever state is required to produce the right HTML and include it within "x")
2) Use that state within the initialize function to build up the HTML you need.
But thinking about it further it seems like this won't avoid the problem you have as it doesn't allow pandoc to do the base64 encoding. We have the attachments API but I don't think that would allow pandoc to do the encoding either (@jcheng5 and @ramnathv correct me if I'm wrong).
It looks like we've got "x" in hand when we call the custom html function:
@jcheng5 and @ramnathv Is there any reason for us not to pass "x" as well here?
If I remember correctly, we refactored the *_html
functions so that they don't take x
as an argument. This was done since x
may not be available upfront, while using widgets in the context of a shiny
application.
One way to make this work with the attachments API is to provide a base64 encoding function on the javascript side, instead of letting pandoc
handle it. Not sure how feasible it is and if it would deteriorate performance, but something to think about. @jcheng5 thoughts?
@ramnathv Sorry, I didn't understand your suggestion.
@dmurdoch This isn't a solution, but, does it slow down even if you have <img src="data-uri">
rather than <img src="file">
? I just tried embedding an 850K character img src into an R Markdown doc and it didn't seem to slow down.
@jcheng5 scratch that solution. it doesn't solve the issue at hand.
@jcheng5 : I haven't tried using HTML <img src="data-uri">
; I was putting the data-uri
into a field in x, so it would be sent as part of the JSON string, and Javascript code attached it to an image before display. But I would have guessed your example would be just as slow, so I might be completely wrong about the cause of the slowdown. I'll put together some simpler examples and see if I can work out the pattern.
Okay, I've done some testing. You can get the files I used from http://www.stats.uwo.ca/faculty/murdoch/temp/base64. The image file is 2.9 Meg, which is quite a bit bigger than @jcheng5's example, so maybe that makes the difference.
When I run
system.time(rmarkdown::render("imageInFile.Rmd")) # use src="file"
system.time(rmarkdown::render("imageInURI.Rmd")) # use src="data-uri"
system.time(rmarkdown::render("imageInString.Rmd")) # build the image in Javascript from a data-uri string
I get times of 0.3, 13, and 7 seconds respectively. The Rmd to md step takes longer with the bigger files in the 2nd and 3rd cases, but I think most of the time is spent in pandoc.
OK, I get the same times as you with your test cases.
If pandoc weren't written in Haskell I'd be tempted to dive into their code and just figure out why it's so darn slow with these big strings.
OK it does seem to just be a linear slowness based on size of input. I erroneously thought the slowness was specific to long strings without newlines, but injecting newlines every 100 characters doesn't help at all.
Really, this is what the attachment feature is for, it's a shame it doesn't work with pandoc. Maybe if we reimplemented it using img tags?
I don't know the attachment feature.
Something else that might solve my problem would be to write out the JSON string to a separate file, and have the *.md file use <script type="application/json" data-for="foo" src="jsonfile"></script>
instead of including it inline. I don't know if that's something I could do with a custom JSON serializer, I'll take a look.
The rglwidget() now does a nice display in the Viewer pane from within RStudio, good enough that I might want rgl to default its display there. The problem is the incremental building of a display.
In Shiny I can use a custom message to send updates (as recommended by you guys at the end of August, but it took me a while to realize what a good solution it was). However, those messages only work in Shiny.
It would be good enough for my purposes to just redraw the whole thing after a change, but I don't want all the incremental versions to show up in the history.
Another idea I had was to make a simple Shiny app with no server running. Perhaps a function call in R could establish a connection with the display in the viewer, send a custom message, then shut down the connection and return control to the user.
Any suggestions how to allow R code to update the viewer without adding to the history?
Within the RStudio IDE's implementation of the Viewer pane you are served off of the same port as the R help server, so if you register a custom URL handler for the help server you'd be able to address it from within your widget's JavaScript.
That said, this only applies to the RStudio IDE Viewer not to Shiny or R Markdown documents and not to widgets that have been saved as HTML.
In terms of detecting when you are in the RStudio Viewer, we append a "viewer_pane=1" query parameter whenever we serve a widget within the Viewer.
On Fri, Oct 9, 2015 at 7:56 AM, dmurdoch notifications@github.com wrote:
The rglwidget() now does a nice display in the Viewer pane from within RStudio, good enough that I might want rgl to default its display there. The problem is the incremental building of a display.
In Shiny I can use a custom message to send updates (as recommended by you guys at the end of August, but it took me a while to realize what a good solution it was). However, those messages only work in Shiny.
It would be good enough for my purposes to just redraw the whole thing after a change, but I don't want all the incremental versions to show up in the history.
Another idea I had was to make a simple Shiny app with no server running. Perhaps a function call in R could establish a connection with the display in the viewer, send a custom message, then shut down the connection and return control to the user.
Any suggestions how to allow R code to update the viewer without adding to the history?
— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/141#issuecomment-146842561 .
I don't understand how to "register a custom URL handler for the help server". Is that something that can be done from R code?
When Simon first added the help server in R 2.10 there was an environment within the tools package that you could add a named function to which would cause an HTTP handler to be registered at e.g. "/custom/foo". Here is a reference to this mechanism in the docs of the Rook package:
These handlers are served off of the help port (which we also use for serving htmlwidgets in the viewer).
I'm not sure though whether this mechanism was closed off in R 3.2 (there were some other changes to the help server in that release). Simon will know for sure.
On Fri, Oct 9, 2015 at 2:23 PM, dmurdoch notifications@github.com wrote:
I don't understand how to "register a custom URL handler for the help server". Is that something that can be done from R code?
— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/141#issuecomment-146953902 .
In a knitr document, I may have several variations on the same rgl scene. Using the old writeWebGL scheme I could figure out the name of the early one and copy some data from it in the later ones. I'd like to do the same with the rglwidget implementation. I can do it by explicitly specifying the elementId of each widget, but it would be nicer if the code could work out the auto-generated names that knitr will use.
Is there a way to figure out either the next auto-generated name, or the last one that was used?
I think that the explicit elementId is currently your only recourse (here's where we determine the elementId if it's not provided: https://github.com/ramnathv/htmlwidgets/blob/master/R/htmlwidgets.R#L52).
It wouldn't be too difficult to provide a bit more transparency here by pre-generating blocks of ids on demand (thus making available nextId, previousId, etc.) however this would be fragile visa vi whether your widgets were interspersed with other widgets.
On Sat, Oct 17, 2015 at 12:32 PM, dmurdoch notifications@github.com wrote:
In a knitr document, I may have several variations on the same rgl scene. Using the old writeWebGL scheme I could figure out the name of the early one and copy some data from it in the later ones. I'd like to do the same with the rglwidget implementation. I can do it by explicitly specifying the elementId of each widget, but it would be nicer if the code could work out the auto-generated names that knitr will use.
Is there a way to figure out either the next auto-generated name, or the last one that was used?
— Reply to this email directly or view it on GitHub https://github.com/ramnathv/htmlwidgets/issues/141#issuecomment-148929753 .
I tried setting the elementId automatically, but then Shiny gives a warning. Is there a way to detect that my rglwidget() is being called from a Shiny server?
Are you generating elementId automatically on the client side (in javascript?). If so, you can check for the presence of window.Shiny
to detect that the widget is being called from a shiny server.
I'd like to make an rgl widget, but I'm having trouble getting started, because the examples I've seen start with a Javascript library, rather than starting with R code. Here's what I'm thinking of doing:
Does this sound like the right approach? Are there any examples to work from that are similar?
Duncan Murdoch