Graphics from remote sessions

[lawremi]

The inconvenience of viewing plots produced in a remote session is a major deficiency of ESS, especially given the popularity of working on compute clusters and the cloud. I think this can be fixed. Right now, it's easy to capture PNG data (as a raw vector) using the Cairo and png packages. Emacs can obviously display PNGs. How hard would it be to send the PNG data over some sort of channel from R to Emacs? Could we just use console output that is hidden from the user?

With some simple improvements to the grDevices package, we can stream PNG, PDF and SVG data to R connections. I can make this change, but it won't be part of a release for another year, so the Cairo and png packages should be used for prototyping.

[vspinu]

The inconvenience of viewing plots produced in a remote session is a major deficiency of ESS,

What is the inconvenience more concretely? If you have a fast remote, X11 forwarding should do a decent job.

Could we just use console output that is hidden from the user?

The main difficulty is the lack of bidirectional MPI in ESS. ESS was perceived as a simple system and it has its advantages because of that. There have been recent changes to ESS that allowed a simple MPI for streaming messages through the standard ouput. This will take care eventually of images, hopefully by next release.

Right now, it's easy to capture PNG data (as a raw vector) using the Cairo and png packages.

Capturing image data could be done with plain R. The trick was to capture incremental changes to plots, but last time I have looked at it it was doable within plain R graphics framework as well.

[eddelbuettel]

@vspinu A not-so-uncommon use case is outside of X11. (I don't work that way but some people do, or sometimes do.) And Ryan @hafen has expressed this well -- and has new tools: initial release / post, followup, github repo. Given how emacs can merrily display graphics file formats, I see this as a potential winner too.

[lawremi]

What is the inconvenience more concretely? If you have a fast remote, X11 forwarding should do a decent job.

Not decently enough in practice. X11 forwarding is slow and breaks e.g. when detaching a screen session.

Capturing image data could be done with plain R. The trick was to capture incremental changes to plots, but last time I have looked at it it was doable within plain R graphics framework as well.

I could see doing it using temporary files, but I'm not aware of a way to capture graphics in a format that Emacs would understand, completely in memory. Maybe I just have an irrational distaste for touching the file system when it is avoidable. As discussed in a thread on ess-help, the evaluate package demonstrates one way to capture plot updates.

Thanks @eddelbuettel for pointing out the rmote package. Outside of embedding in an Emacs buffer (not a strict requirement), it seems to solve the problem. I like how it handles htmlwidgets in addition to R graphics. I was looking into how to do that with ESS, as well, and was disappointed to find that Emacs lacks webkit integration (outside of a hack involving PyQt).

@hafen mentioned the need to override print methods. I think that could be solved more cleanly using trace(), but there is also the problem of calling plot_done() with base graphics. Both problems I think could be solved using addTaskCallback() to respond to each evaluation, and then mimic (or ideally depend on) what the evaluate package does to detect changes in the graphics device.

[hafen]

@lawremi, I think it would be possible to get around the print method hacking - IIRC @gmbecker or @gaborcsardi had some ideas along the same lines as what you suggested. I just needed a quick solution as I've been buried with consulting work lately that requires this. I'm interested polishing up the print intercept - if you want to discuss more, I've opened up an issue here.

[lionel-]

Outside of embedding in an Emacs buffer (not a strict requirement)

Emacs 25 can now embed Webkit widgets inside buffers, so there should be a way to do this.

@hafen mentioned the need to override print methods. I think that could be solved more cleanly using trace(), but there is also the problem of calling plot_done() with base graphics. Both problems I think could be solved using addTaskCallback()

I don't know a lot about this problem, but wouldn't the cleanest solution involve writing a custom graphics device?

[lawremi]

Emacs 25 can now embed Webkit widgets inside buffers, so there should be a way to do this.

Cool to hear about the XWidget branch being merged. Missed that. Now we can even embed GGobi plots...

wouldn't the cleanest solution involve writing a custom graphics device?

Writing a custom graphics device is a lot of work. Perhaps the implementation could be simplified by delegating to another device to do the actual rendering, but that highlights a design issue. Rendering graphics is orthogonal to capturing a plot. Depending on the use case, we might want to capture the source (like the ggplot object), the rendered form or both, but it often does not matter how the plot was actually rendered. Ideally, we could achieve for the R console what the evaluate package has done for batch evaluation. The evaluate package relies on a lot of hacks, so maybe something could be done in the R internals to make all of this easier.