Modular Rendering

[nx10]

This is a tracking issue for the modular rendering branch.

This will resolve:

• [x] Image formats nx10/httpgd#58
• [x] Client node preprocessing pipeline nx10/httpgd#68
• [x] Client export image dialog nx10/httpgd#59 (in progress)
• [ ] Copy to clipboard not working on all platforms nx10/httpgd#49
• [ ] svgz support nx10/httpgd#42
• [x] External CSS rendering mode nx10/httpgd#65

ToDo:

• [x] Link with cairo library
• [x] Add binary HTTP response support to belle library
• [x] Implement modular rendering
  • [x] HTTP API
  • [x] R API
• [x] Server side zoom
• [x] Make cairo system dependency optional on linux and macOS
• [x] Update API documentations
• [ ] Testing
• [ ] Renderers:
  • [x] SVG
  • [x] Portable SVG (#42)
  • [x] Cairo
  • [x] PNG
  • [x] PDF
  • [x] PS / EPS
  • [ ] JPEG
  • [ ] Uncompressed bitmap formats (TIFF / BMP / ...) (?)
  • [x] TikZ ~- [ ] xfig (is this still used? base R supports it)~
  • [x] JSON (partly done)
  • [x] Special renderers:
  • [x] Plot strings (for searching)
  • [x] Plot metrics (number of draw calls, ...)

The goal of this is for httpgd to become a complete network accessible replacement for all R graphics backends. (And eventually be faster if possible.)

Preview:

[ManuelHentschel]

Would it be somehow possible to produce "annotated SVG" plots? I.e. include information about data points as attributes in their corresponding SVG elements? This would allow some very useful features like hovers showing the coordinates of a point (as e.g. in excel graphs). I'd assume that this is rather difficult due to the low-level nature of plotting devices, but if there's any chance of implementing this, I'd be happy to add frontend-support in vscode or help otherwise :)

Also, would a "generic device" renderer make any sense? Currently, I am using httpgd to "fine-tune" my plots and then switch e.g. to a tikz device to use the plot in latex. This works ok, but can be a bit annoying, e.g. if I produce lots of plots and later choose the best one, or if the plot-function contains some randomness (like e.g. some graph-plotting functions). I guess my current usecase will be covered by Tikz renderer on your todo list, but for the general case, this might be a useful feature. And this approach might save you from implementing some less popular renderers (xfig??) yourself.

[nx10]

Annotated/interactive plots would be very cool, but I am afraid this is not possible with the current graphics information R passes to devices. This would need additions to the R graphics engine itself. Maybe this could be proposed to the R core team?

Regarding your "generic device": with one of the last commits I made to the modular rendering branch I added a JSON renderer that completely provides all plot information in a machine readable way. Does this help you?

Edit: Regarding xfig: I had a look at the website of this software and it seems like it has not been updated in many years. Also as far as I can tell it can import SVG and other formats, so I will skip the support unless anyone shows strong interest in it. I am honestly surprised this is still supported by base R.

[ManuelHentschel]

Regarding your "generic device": with one of the last commits I made to the modular rendering branch I added a JSON renderer that completely provides all plot information in a machine readable way. Does this help you?

Yes, kind of. In order to create a Tikz plot of the exported plot, I'd need to write a function that parses the JSON, opens a Tikz device, and then makes a corresponding clip/draw-call for each entry in the JSON. Is that correct? If yes, would it be feasible to integrate this in httpgd so that I only need to call e.g.

httpgd::hgd_add_device_renderer(tikzDevice::tikz(filename, ...))

and would then have the option to export a plot to that device?

(I guess my personal usecase will be covered by the Tikz renderer, but maybe this approach would work to support other output formats without having to reimplement all the renderers?)

[ManuelHentschel]

Annotated/interactive plots would be very cool, but I am afraid this is not possible with the current graphics information R passes to devices. This would need additions to the R graphics engine itself. Maybe this could be proposed to the R core team?

I'd assume that this is too much of a change to the graphics engine to be considered by the core team (considering how important backwards compatibility etc. is in R).

A somewhat hacky workaround might work using a plot function like

myPlot <- function(x, y){
    ind <- seq_along(x)
    cl <- sapply(ind, function(i) rgb(0, 0, i, maxColorValue = 255))
    plot(x, y, col=cl)
    tbl <- data.frame(
        color=cl,
        x=x,
        y=y
    )
    print(tbl)
}

x <- 1:5
y <- x**2
myPlot(x, y)

and then have a front-end that adds the hover and corrects the colors using javascript. The info in tbl would need to be embedded in the plot somehow, e.g. using the custom css parameter or a plot label etc.

[nx10]

The problem with that approach is, that there is no information about margins, axes, etc. in the data so there is no clean way to do this. Additions to the graphics engine are usually done with capability flags to provide backwards compatibility.

[nx10]

Yes, kind of. In order to create a Tikz plot of the exported plot, I'd need to write a function that parses the JSON, opens a Tikz device, and then makes a corresponding clip/draw-call for each entry in the JSON. Is that correct?

Now I understand your original question better. I am not sure how good this will work. A bit more info on how httpgd/graphics devices work with R:

Each time a plot changes it's resolution httpgd saves a pointer to a plot snapshot. These pointers can be used to replay a specific plot after adjusting the output resolution. You can not access the information inside them, and I imagine they contain callbacks to actual R code so it's probably not feasible to do so. All graphics devices will then just receive callbacks to primitive drawing function they provide (e.g. draw a line, circle, rectangle, ...) and use them to generate a graphic in some way. httpgd has an intermediate in-memory representation of that data. (What you can see in the JSON output.)

So the JSON does include all the static information the device has access to, but not the dynamic. So you will loose the capability to resize the plot properly. I would suggest waiting for the tikz renderer, or, if randomness is your primary problem considering set.seed to control it.

Edit: if you want to draft a proposal I can help you. I do think annotations could be a useful addition.

[ManuelHentschel]

Each time a plot changes it's resolution httpgd saves a pointer to a plot snapshot. These pointers can be used to replay a specific plot after adjusting the output resolution. You can not access the information inside them, and I imagine they contain callbacks to actual R code so it's probably not feasible to do so.

Does this "replay" happen inside the httpgd package or is it done by R itself and httpgd just receives the primitive drawing function calls?

All graphics devices will then just receive callbacks to primitive drawing function they provide (e.g. draw a line, circle, rectangle, ...) and use them to generate a graphic in some way. httpgd has an intermediate in-memory representation of that data. (What you can see in the JSON output.)

Would it in principle be possible to record the exact content of these callbacks and later send them to a different graphics device? Or are there differences between the callbacks depending on the individual device?

[nx10]

Does this "replay" happen inside the httpgd package or is it done by R itself and httpgd just receives the primitive drawing function calls?

It happens inside R, and as far as I can tell depends on the environment, loaded packages, etc.

Would it in principle be possible to record the exact content of these callbacks and later send them to a different graphics device? Or are there differences between the callbacks depending on the individual device?

I think I saw a function in the R API that lets you serialize them but I don't know if they will always be compatible between different devices. Maybe you can try to play around with their R bindings: grDevices::recordPlot. These R/C APIs are a bit obscure and I learned most of the things I know by actually looking at the source code of R and the base packages. I imagine this is why ggplot2 added their own intermediate representation before the R APIs (grob), which is easily serializible from within R.

[ManuelHentschel]

Thanks for the clarification! Recording/replaying plots seems to be a bit harder than I imagined...

These R/C APIs are a bit obscure [...]

I noticed that, which is why I only have a very vague idea how the graphics devices work. I might play around with recordPlot a bit, but since this sounds somewhat complicated and I would only use this for tikz at the moment, I don't think I'll actually write anything useful.

I'll give annotated SVGs a shot though!

[nx10]

Recording/replaying plots seems to be a bit harder than I imagined...

Yes that's why one of my main goals with httpgd is abstracting all of this complexity away.

I'll give annotated SVGs a shot though!

Exciting! Let me know how this works out.

[nx10]

@ManuelHentschel Just to let you know: I have added a first, experimental version of a TikZ renderer eca26139006ad18b75aaa1230ae6c98f7ac0f7df . TikZ rendering has some unique challenges, let me know if you find any graphical errors. There is a lot of room for optimization in the future. Fonts and rasterImage() draw calls are not yet supported.

As with SVG+HTML, this lets you plot for RMarkdown in memory using TikZ/LaTex+PDF, which is useful for quick testing:

---
title: "RMD test"
header-includes:
   - \usepackage{tikz}
output: pdf_document
---

```{r, include=knitr::is_latex_output()}
s <- httpgd::hgd_inline({
  hist(rnorm(100))
}, width=400, height=300, renderer="tikz")
knitr::asis_output(s)
```

[nx10]

I have released httpgd 1.2.0 to CRAN. More renderers (JPEG, TIFF, SVGZ) will follow in a later update.

[nx10]

Closing as modular rendering is implemented and released. Missing renderers will be tracked in nx10/httpgd#42 and nx10/unigd#3 .