Support jupyter-console with ess-remote

jackkamm commented 5 years ago

I would like to connect ESS with a running jupyter R kernel, by executing jupyter console --existing in a *shell* buffer, and then calling M-x ess-remote. I encounter these problems: 1) ess-remote manages to set the *shell* buffer to inferior ESS mode, but then freezes until I hit C-g. 2) ESS does not recognize the jupyter console prompt, which looks like In [86]: instead of >. This breaks cycling through the comint history among other things, and is possibly the root cause of (1) above (the inferior ESS may be hanging because it can't find the prompt). 3) Using C-c C-c to send paragraphs to the inferior ESS doesn't work. The statements are sent to the inferior ESS, but appear not to be executed. However C-c C-j (ess-eval-line) and C-c C-f (ess-eval-function) appear to work fine.

I think the first step is to get inferior ESS to recognize the jupyter prompt, as this may help solve the other issues. I've tried setting inferior-ess-primary-prompt but it seems to have no effect (and is reset by ess-remote anyways).

My motivation for all this is to run R in a docker container on a remote server, use it with org-mode via ein (for its nice handling of async execution and remote sessions), but still be able to use ESS to evaluate statements while editing in the Org Src buffer.

Please let me know if there's any more info that would be useful, or if you have any advice/short-term workarounds/etc. Would also be happy to help with a PR when we've figured out how to solve this...

jabranham commented 5 years ago

Prompt detection is pretty essential to inferior buffers, and it's not at all straightforward to change. Can you tell the jupyter R to use > as its prompt?

jackkamm commented 5 years ago

Unfortunately it doesn't look like there's a good way to change the jupyter prompt. It may be possible with some hacks, but it's not really clear -- here are a couple links for future reference: https://github.com/ipython/ipython/wiki/Cookbook:-Dynamic-prompt https://newton.cx/~peter/howto/extend-prompt-in-ipython/

vspinu commented 5 years ago

Do you really need jupyter notebook? Dockerised R can be run happily with docker-tramp.el and ESS.

Changing prompt on ESS is challenging. Just too many things depend on it.

jackkamm commented 5 years ago

While tramp can support remote/docker R sessions, they aren't persistent -- if I close emacs or lose the connection, then the session is lost.

IMO the main advantages of jupyter are:

Persistent remote sessions
Asynchronous evaluation of code blocks (it doesn't hang when evaluating a computationally intensive block)

I like org-babel as an alternative to jupyter/Rstudio notebooks, but these 2 features are important to me. ob-ein solves this nicely, by letting me use jupyter with org-mode.

I've explored various alternatives to jupyter over time, but they each have some downsides, here's a list:

For persistent sessions, I can run emacs remotely in tmux, but then I can't view graphics in emacs. Other pain points are X11 forwarding in tmux, and installing my emacs config on every remote machine.
Another alternative is to run R with dtach and use ess-remote. This works decently, but still some bugs, for example ob-R will dump ugly ^M symbols at newlines in my RESULTS blocks.
For org-mode asynchronous evaluation, I can separately evaluate long running computations in the inferior ESS, then create a separate SRC block to print the results in the org-buffer. This gets the job done, but is a little awkward.
There is an ob-async package for asynchronous org-mode evaluation. But it doesn't support interactive sessions -- it will launch a new R process for evaluating each block.

For my immediate needs, using ob-R with some combination of the workarounds in the above list, works. But I would like to find a way to improve the situation for persistent, remote, interactive notebooks in emacs, and I think jupyter has the right architecture for this. ob-ein already has some support for code-editing with ESS, but it would be great if we could find a way for it to work with inferior ESS as well.

A potential alternative to jupyter would be if emacs could connect with a remote Rstudio server session. I have no idea if this is possible, but it might play better with ESS than jupyter.

jackkamm commented 5 years ago

Perhaps a solution to support jupyter in ESS, would be to add a new ess-dialect for jupyter, and set its prompt to be \\[\\d+\\]]:, as well as setting the inferior-ess command to some variant of jupyter console --existing. I've been experimenting with this a little bit, with mixed success.

vspinu commented 5 years ago

I have started recently thinking of a design of ess-connect where ESS could connect to running R sessions through tcp, websockets and ssh without much ado. My original hope was to use screen or tmux but both of those are too much pain to integrate without a proper terminal emulation. I didn't know about dtach. It might be the way to go for a simple version given current ESS internals. If ^M is the only problem those are easy to deal with. To allow for proper asynchronous evaluation a new way of handling IO would be needed though.

jackkamm commented 5 years ago

Improving the ESS+dtach integration would be really great, when I have some time I'll try to create a reproducible example of some of the issues I was having before, and file a separate issue. (If you do wind up working on the websocket stuff I would also be interested to know about it and try to be helpful).

dcalacci commented 5 years ago

just came across this looking for something else, but for folks who are interested in dtach + ess, I wrote a little function a while ago that works pretty well. Source files are local but it's all run on a remote machine. details are sort of documented in a post on my website.. not the same as using jupyter-console obviously but using this I haven't had any of the issues mentioned

(defvar R-remote-host "your-remote-server")
(defvar R-remote-session "R-session-name")
(defvar R-remote-directory "/path/to/your/project/directory")
(defun R-remote (&optional remote-host session directory)
  "Connect to the remote-host's dtach session running R."
  (interactive (list
                (read-from-minibuffer "R remote host: " R-remote-host)
                (read-from-minibuffer "R remote session: " R-remote-session)
                (read-from-minibuffer "R remote directory: " R-remote-directory)))
  (pop-to-buffer (make-comint (concat "remote-" session)
                              "ssh" nil "-Y" "-C" "-t" remote-host
                              "cd" directory ";"
                              "dtach" "-A" (concat ".dtach-" session)
                              "-z" "-E" "-r" "none"
                              inferior-R-program-name "--no-readline"
                              inferior-R-args))
  (ess-remote (process-name (get-buffer-process (current-buffer))) "R")
  (setq comint-process-echoes t))

vspinu commented 5 years ago

ess-remote is a bit clumsy, let's try putting together a simple remote connector with full ESS support #968

emacs-ess / ESS

Support jupyter-console with ess-remote #778