Feature request: Display running status of `ob-jupyter`

[Yevgnen]

When using ob-jupyter with :async, I find that quite often I get lost when I eval all blocks or blocks before point. When waiting for the async results, I usually do some other editing. But it's hard for me to check which block is running or whether it has finished.

I tried to use hooks like org-babel-after-execute-hook to start and stop a spinner to indicate the evaluation progress. But it failed in the :async mode.

I also tried to use a counter to advice jupyter-generate-request or jupyter-handle-execute-reply since I found that a reliable way to check whether a block has finished is in jupyter-handle-execute-reply as I noticed the message "Code block evaluation complete." was from there.

(cl-defmethod jupyter-handle-execute-reply :before ((_client jupyter-org-client)
                                                        (req jupyter-org-request)
                                                        status
                                                        _execution-count
                                                        _user-expressions
                                                        payload)
      (decf jupyter-org-bable-request-count)
      (if (= jupyter-org-bable-request-count 0)
          (jupyter-org-bable-reset-request-count)))

But it failed. jupyter-generate-request seems not the right place to start the counter. And I know very little about cl things or concepts (like cl-defmethod).

It will be really nice to have functions to

• go to current running block
• show running status in modeline
• display the number of pending requests and the total requests [3/11] of last evaluation request (like org-babel-execute-buffer, jupyter-org-execute-to-point) in modeline or somewhere
• stop all pending requests (jupyter-repl-interrupt-kernel seems only stop the current running one)

How do you think about it?

---

By the way, thanks for the great package. I've been tried to get into org-babel in the last eight years. But failed even after I tried org-babel itself, ein, and ob-ipython and scimax, many times. I try emacs-jupyter started from this week and it is super robust and the experience is really great!

[nnicandro]

• go to current running block

How would this work? Although its not very common, at least in my use case, there can be multiple source blocks talking to different kernels running at the same time in a single document. Which one would be the current running block? I suppose we can limit it to the kernel for the source block at point.

• show running status in modeline

Do you mean having some indicator for when the kernel is busy in the org-mode buffer like the indicator in the REPL (see jupyter-repl-interaction-mode-line)? Again, there can be many kernels running in one document. Maybe we could adapt jupyter-org-closest-jupyter-language to get the closest kernel client and show the status of that one. So we would have something similar to which-function-mode but for the status of a kernel.

• display the number of pending requests and the total requests [3/11] of last evaluation request (like org-babel-execute-buffer, jupyter-org-execute-to-point) in modeline or somewhere

I had previously thought of doing something like the Jupyter notebook which displays [*] next to cells that are pending or haven't received an :execute-reply yet. We could place [*] on all pending source blocks as an overlay on the #+begin_src line.

So it would look like, with some distinguishing face of course,

#+begin_src jupyter-python :async t [*]
1 + 1
#+end_src

This way if partial results are received by the kernel, you still have an indicator that the source block is still running. Does this sound like what you are looking for?

• stop all pending requests (jupyter-repl-interrupt-kernel seems only stop the current running one)

This would require a change to how we send requests since there is no way on the kernel side to stop pending requests. Sending an interrupt request to the kernel will only interrupt the current computation, as you have seen, and the kernel will just start a new computation from the queued requests. There isn't a way to say "clear all queued requests" on the kernel side.

We could possibly do the queuing on the Emacs side. For example, only sending an :execute-request after a previous :execute-request has received an :execute-reply. Doing this would also help in having indicators for the currently completed/queued source blocks.

By the way, thanks for the great package

Your welcome 👍.

[Yevgnen]

there can be multiple source blocks talking to different kernels running at the same time in a single document.

Oh, I didn't think about that! I've never run different kernels in a single document, even in the browser with jupyter notebook. The power of the org mode makes that possible.

I suppose we can limit it to the kernel for the source block at point.

I have another idea. Would just jumping to a block of 'some busy kernels' or cycle through block 'some busy kernels' works better? In this case, we don't need to care about where the point is. But I don't know checking whether a/some kernel/s is busy is time consumed or not. I have little knowledge about that...

Do you mean having some indicator for when the kernel is busy in the org-mode buffer like the indicator in the REPL

This way if partial results are received by the kernel, you still have an indicator that the source block is still running. Does this sound like what you are looking for?

Yes, and yes.

But I personally find the Jupyter notebook way [*] is not that useful for me. Still when I eval all the blocks (100+) of a buffer which may takes minutes, I may adding new contents at the same time. It's not easy for me to check the progress. I may have to search through to see which async id has been removed many times (the same faces for a block and its result have annoyed me). Without any indicator, I don't even know whether an error has been raised.

Maybe I can search the [*] pattern, to find the busy block, and check the percentage of progress. But if there's command that can jump to 'some busy/errored block' as mentioned above, and some indicator like [3/11] showing the unfinished/finished number in mode line will be more handy and intuitive. In the multiple kernels situation, maybe we can show indicators like Python 3[3/11] and R[-] and Julia[error] periodically for every kernel?

When evaluating a whole buffer, without the ability to stop all pending requests usually cause many exceptions in all the follow blocks...The idea of queuing in the client side sounds reasonable. 👍

[nnicandro]

Would just jumping to a block of 'some busy kernels' or cycle through block 'some busy kernels'

Good idea.

I don't know checking whether a/some kernel/s is busy is time consumed or not.

No its not. The Jupyter messaging spec always sends a status: busy message once an :execute-request has started and sends a status: idle message once its done. Whenever we receive such messages the state of various objects are updated so its just a matter of checking the properties of those objects.

You can check if a kernel is busy using the jupyter-kernel-busy-p function and if a certain request object has been completed already using jupyter-request-idle-received-p or jupyter-org-request-idle-received-p for jupyter-org-request objects.

It's not very easy to get at this information in a convenient way right now, but I will make it easier to do so and add some jupyter-org-(next|previous)-busy-src-block functions.

maybe we can show indicators like Python 3[3/11] and R[-] and Julia[error] periodically for every kernel?

In my opinion this would be too distracting. I'm thinking we just have [evaluated/queued] for the entire file even with multiple kernels, but I'm not leaning in either direction at the moment.

How would we get these counts though? We can get a queued count whenever jupyter-org-execute-subtree (or similar) is called and then increment the evaluated count whenever we send an execute request from the queued requests, but what if another jupyter-org-execute-subtree is called in the meantime? Do we just add to the current queued count?

queuing in the client side

What should be queued on the client side though? There are two options (1) save a marker to the source block and just call org-babel-execute-src-block when its time to execute it or (2) take a snapshot of the necessary state whenever org-babel-execute:jupyter is called (marker and arguments) and use that when its time to execute the source block.

If we go with (1) that means the user can modify the source block before it gets evaluated whereas (2) means that only the contents of the source block at the time jupyter-org-execute-subtree (or similar) is called will be used.

The current implementation is essentially (2), but it would be simpler to go with (1). I'm still thinking about which one makes more sense though.

[Yevgnen]

Do we just add to the current queued count?

My original thought is, just to add it up in the single kernel situation. I only care about [unfinished/finished]. If the multiple kernels indicator is a bit distracting, maybe we can just degrade everything to [idle] and [busy] or ○ and ● (the jupyter notebook takes this) in the mode line regardless of the number of kernel to keep things easy and simple. 😆

I tested the evaluation behavior in the jupyter notebook, it's (2). Maybe keeping the same behavior is better.

[nnicandro]

I've added some functions jupyter-org-(next|previous)-busy-block and, if you have hydra installed, set them as the N and P bindings in the Hydra accessible by pressing C-c h in org-mode. See 653f934bfc3027bfb57a8e10b0069a087e747890.

[Yevgnen]

Thanks. After a quick try, the functions look good. Now I have to admit that the [*] for each cell is also helpful. 😅