ob-async should support :session header-arg

[astahlman]

Desired behavior

#+BEGIN_SRC sh :session foo :async
export GREETING="Hola"
#+END_SRC

#+RESULTS:

#+BEGIN_SRC sh :session foo :async :results output raw
[[ -n "$GREETING" ]] && echo "$GREETING" || echo ":session is not respected"
#+END_SRC

#+RESULTS:
Hola

Current behavior

#+BEGIN_SRC sh :session foo :async
export GREETING="Hola"
#+END_SRC

#+RESULTS:

#+BEGIN_SRC sh :session foo :async :results output raw
[[ -n "$GREETING" ]] && echo "$GREETING" || echo ":session is not respected"
#+END_SRC

#+RESULTS:

:session is not respected

[stsquad]

Can async support this? I thought :session was meant to run things in a inferior process which should be asynchronous anyway?

[drym3r]

Can async support this? I thought :session was meant to run things in a inferior process which should be asynchronous anyway?

No, it doesn't. emacs-async opens an independent emacs process which executes all that's in the block. What could be done, maybe, is writing all the evaluated stuff and every time something is evaluated, that file is readed and evaluated on the independent emacs process. Something like this workflow:

I execute this:

+BEGIN_SRC sh :session foo :async

export GREETING="Hola"

+END_SRC

The independent emacs process evaluates it and writes the result on /tmp/something/foo, for example. Then, when we execute this:

+BEGIN_SRC sh :session foo :async :results output raw

[[ -n "$GREETING" ]] && echo "$GREETING" || echo ":session is not respected"

+END_SRC

Before executing it, /tmp/something/foo could be sourced. I guess it's not that easy and maybe it won't be usefull in all cases or languages, but maybe in some yes.

[astahlman]

Hmmm, yeah - that's a good point. If we were to serialize the session to a file and then source it in an independent process then the contents of the blocks would need to be idempotent.

I wonder if it would be possible to start a persistent session in the child process with async-start and send it the contents of src blocks for evaluation with async-send.

[drym3r]

That may be a good way to do it. That's how python or bash :session works (I believe), it creates a background process and feeds them with all the commands.

[astahlman]

I wonder if a thread-based implementation (supported in Emacs 26) would make this easier? See this comment for details.

[preetpalS]

This following comment was based on the belief that Emacs 26 would allow multiple threads to execute at once (this is not the case). See discussion following https://github.com/jwiegley/emacs-async/issues/97#issuecomment-365859069.

@astahlman I'm not completely familiar with the semantics of sessions (I am assuming they are only for interpreted languages (except Emacs Lisp? See ob-emacs-lisp.el in https://code.orgmode.org/bzg/org-mode/src/master/lisp)) but I believe that you could view sessions as being uniquely accessed using a composite key consisting of the session name and programming language.

There are two paths (that I can think of right now) that you could take to implement sessions:

1. Integrate with existing sessions support in upstream org babel supported languages.
2. Independently support sessions (this would be confusing since sessions of the same name would not be shared between async and synchronous code blocks).

The following discussion, assumes option 1 is the only viable option.

Assuming that sessions are lazily initialized (see https://code.orgmode.org/bzg/org-mode/src/master/lisp/ob-ruby.el#L109 ; for ob-ruby they are), you would probably need mutexes to guard against concurrent initialization of sessions. You would also probably need to synchronize access to the sessions also. There might be issues if sessions are currently being created in a way that any interactions with them block the main thread. I don't think that this library can integrate sessions without coordinating with upstream.

[astahlman]

@preetpalS Thanks for looking into it, any rate. The discussion in https://github.com/jwiegley/emacs-async/issues/97 was enlightening.

[fkgruber]

For R code blocks the :session support is just critical. I can't see much use otherwise. Anyway fantastic for shell blocks and others.

[nico202]

Any luck @jkitchin is interested in helping in this? :pray_tone:

[jkitchin]

This might already work with ob-ipython (which can support other kernels via Jupyter). In the scimax modifications to ob-ipython (https://github.com/jkitchin/scimax/blob/master/scimax-org-babel-ipython-upstream.el) there is support for async execution that works for ipython, but I guess should work for R too, with an R jupyter kernel. I haven't used other kernels very extensively though, so I don't know if there will be issues with it.

[jackkamm]

I've started work on this upstream: http://lists.gnu.org/archive/html/emacs-orgmode/2019-06/msg00014.html I've put my branch on github here: https://github.com/jackkamm/org-mode/tree/babel-async-session So far I've added async session eval for R, and added some facilities to extend this to other languages. I believe this feature will require per-language implementation, so would be best done upstream, to facilitate refactoring and reusing existing code. However I have not yet gotten a response whether upstream is interested in this (I just posted last night). If not I'll try to port it over here.

[jackkamm]

Updating my previous comment -- I've moved my code for async session evaluation to a standalone repo at https://github.com/jackkamm/ob-session-async

[suonlight]

@jackkamm ob-session-async is great! I've added async session eval for Ruby https://github.com/jackkamm/ob-session-async/pull/2. It works great for me. :)