ess-command-output buffer

[acid-bbq]

My config is:

• Windows 10
• GNU Emacs 27.1
• ESS 18.10.3
• Org mode 9.4.4
• R 3.6.3

For information I recently changed to this config, so I needed to modify my init.el. Before I used Emacs 24.3.1, Org-mode 7.9.3 and ESS 13.05 without having never faced the issue I describe below.

I use Emacs with org mode and org babel to run R for literate programming to create latex reports. When I run R code from an org file (after having done C-c '), it sends the code to an iESS buffer *R and evaluate it. Fine. But after some code evaluations, a buffer called *ess-command-output pops up. In it is written:

ess-output-delimiter75-END The line I evaluated and its result (it is result first and then line evaluated) ess-output-delimiter75-START

The number 75 can change. If I continue code evaluation, it goes in this buffer but in a messy way. When I go back to the iESS *R buffer, it is mentioned "no process" in the mode line. And thus if I close the the *ess-command-output buffer and evalaute code from the ESS buffer (where the script is written), it asks me to open a new iESS buffer *R:2*.

Quite difficult to reproduce this because I cannot identify when it happens. Can be after few evaluations or more.

Any idea of how this issue can be solved? Thanks.

[lionel-]

It doesn't seem like you're running ESS 18.10.3 but 18.10.3snapshot instead, which is the dev version.

[lionel-]

When I run R code from an org file (after having done C-c '), it sends the code to an iESS buffer R and evaluate it.

Can you please be very specific about the sequence of actions? I see that C-c ' only opens a chunk in a separate buffer. When I send code with C-c C-c, type R code with company completions, or move the cursor around (to get eldoc help in the echo area), I don't see what you're describing.

Also please make sure you are using the most recent dev version.

Can you also confirm that you're not seeing this behaviour when using ESS from a normal R buffer without org involved? Maybe org is not relevant to your report?

[acid-bbq]

Thank you for the prompt reply. Yes I've checked it is 18.10.3snapshot indeed. Do you recommend the non-dev version?

1. I open an org file
2. I hit C-c ' to open a chunk of code. It opens it in a separate buffer
3. In this new buffer ESS[R], I send code with C-c C-n or C-c C-r or C-RET
4. I am asked for R starting project directory. I hit RET.
5. Now the R console opens in an iESS [R] buffer

From there I can work as usual, sending code to the console. I also exit chunks with C-c ', go to other chunks and can still send code the the same process. But at some point in time, when I send code from the ESS buffer, the iESS buffer switch to *ess-command-output* with the code I mentioned above. The first thing I did there was to close *ess-command-output* and try to resend code from the ESS buffer. And this is when it asks me to start a new project directory *R:2*

I have never tried with a normal R buffer. It was always with org mode.

[lionel-]

Do you recommend the non-dev version?

Unfortunately not because the last released version is not compatible with Emacs 27.

Can you please make sure you are running the very last dev version. If not, can you confirm you're still seeing this issue after updating?

[lionel-]

I've seen the described behaviour (out of order output delimiters) while working on the malformed-command branch. In this branch I fixed it like this: https://github.com/emacs-ess/ESS/commit/4e2a9b96dcc2f8a6e120996ecc9d50601454d851#diff-71113682ba755c9cbeb34c109f8e149a8185e2c03353584b5fd29d3db8895a22R1012-R1018

The fix makes sense to me and it implements a pattern recommended in the Emacs documentation for interacting with processes. However I wonder what chain of events leads to the out of order insertion in the first place, and why it was more likely in the malformed-command branch.

[acid-bbq]

Thanks. I installed the version 18.10.3snapshot [elpa: 20210210.1202] from melpa. I will test it and let you know if I experience the same behaviour. But as it is hardly reproducible, I will let you know ASAP if it fixed it.

Not sure what to do with the fix you suggested. Is it implemented or do I have to do something?

[acid-bbq]

And just FYI, when I updated my Emacs config, I first used the all-in-one installation file by Vincent Goulet and got the same issue. So that's why I wanted to try to install Emacs from GNU website and ESS by myself. Actually I just realised that Vincent Goulet file includes Emacs 27.1 with ESS 18.10.2 (so non-dev version?). Is it possible according to your previous comment?

[lionel-]

If you're running Vincent's distribution I guess the MELPA version of ESS (which distributes the dev version) takes precedence?

[acid-bbq]

I'm not using Vincent's distribution. But yes I should try.

[acid-bbq]

So far so good! I just got once a "ess-get-words" buffer, but it did not have any effect on the R process in the iESS buffer. At first sight, it looks like the update solved the issue. I'll keep on testing, but usually the issue should have already happened. I'll keep you informed in any case. Thanks.

[lionel-]

@acid-bbq What is your value of ess-eval-visibly please?

[acid-bbq]

it is "nowait". After reading what it does, I might consider switching to "t".

[lionel-]

nowait is recommended. Visible evaluation hangs Emacs while the input is running.

[acid-bbq]

So I just got *ess-command-output* buffer again. I hit C-<RET> in the ESS buffer to send code to the iESS buffer and it switched to *ess-command-output*. In it I have the ess-output-delimiter END and START... The code is run within it but it's quite strange and the *R* shows no process in the mode-line. Could it be linked to my init.el config?

[lionel-]

Thank you for the update. Could you do me a favour and temporarily add the following to your config file to see if it fixes the issue please?

It should come after loading ESS (otherwise the ESS's definition will overwrite yours).

(defun inferior-ess-ordinary-filter (proc string)
  (ess--if-verbose-write-process-state proc string "ordinary-filter")
  (let ((cmd-delim (process-get proc 'cmd-output-delimiter))
        (cmd-async-restore-alist (process-get proc 'cmd-async-restore-alist))
        (flush (lambda ()
                 (with-current-buffer (process-buffer proc)
                   (save-excursion
                     (let ((marker (process-mark proc)))
                       (goto-char marker)
                       (insert string)
                       (set-marker marker (point))))))))
    (if cmd-delim
        (progn
          (funcall flush)
          (inferior-ess--set-status-sentinel proc (process-buffer proc) cmd-delim)
          (inferior-ess-run-callback proc string))
      (inferior-ess--set-status proc string)
      (inferior-ess-run-callback proc string)
      (funcall flush))
    ;; Restore user process buffer asynchronously as soon as process
    ;; is available
    (when (and cmd-async-restore-alist
               (not (process-get proc 'busy)))
      (ess--command-proc-restore proc cmd-async-restore-alist))))

[acid-bbq]

Thanks. I tried it. I just got the *ess-command-output* buffer. Now in it, I no longer have the *ess-output-delimiter* in it. Just the code I sent from the ESS buffer which triggered it. BUT, I retried to send code from the ESS buffer and it sends it to the current *R* process (thus the iESS remained active and the code sent is not lost). So I am not asked to open a new process *R:2*, which is good.

[lionel-]

Thanks for the throrough descriptions of what's happening, that's helpful.

My working hypothesis is that background commands get cancelled and the process is restored asynchronously. While that's happening, you're sending code to the process which is still hooked to the background task buffer which causes your input to be added to it. Probably ESS pops up the process buffer when input is sent which explains that you see *ess-command-output* popping up.

I think I see how to fix this, I'll try to take some time next week.

[acid-bbq]

OK great, thank you! Just FYI, with the code added to my config file, after the *ess-command-output* and when I resend R code to the console, I have two strange behaviours. First the cursor no longer scrolls line by line but by half page (if I it ctrl key it comes back to normal). Second, in the iESS buffer, when I type (, [, I think keys I assigned skeleton-pair, the cursor goes directly on top of the page (So I don't see the code sent, the cursor is on top with a blank page below.)

[acid-bbq]

Hi, some news about the fix?

[lionel-]

There is no news.

[lionel-]

@acid-bbq Can you please try on the latest dev version once MELPA has been updated?

[acid-bbq]

Thank you. Yes, I will try ASAP.

Do I need to delete the following code you suggested to add in my config?

Thank you for the update. Could you do me a favour and temporarily add the following to your config file to see if it fixes the issue please?

It should come after loading ESS (otherwise the ESS's definition will overwrite yours).

(defun inferior-ess-ordinary-filter (proc string)
  (ess--if-verbose-write-process-state proc string "ordinary-filter")
  (let ((cmd-delim (process-get proc 'cmd-output-delimiter))
        (cmd-async-restore-alist (process-get proc 'cmd-async-restore-alist))
        (flush (lambda ()
                 (with-current-buffer (process-buffer proc)
                   (save-excursion
                     (let ((marker (process-mark proc)))
                       (goto-char marker)
                       (insert string)
                       (set-marker marker (point))))))))
    (if cmd-delim
        (progn
          (funcall flush)
          (inferior-ess--set-status-sentinel proc (process-buffer proc) cmd-delim)
          (inferior-ess-run-callback proc string))
      (inferior-ess--set-status proc string)
      (inferior-ess-run-callback proc string)
      (funcall flush))
    ;; Restore user process buffer asynchronously as soon as process
    ;; is available
    (when (and cmd-async-restore-alist
               (not (process-get proc 'busy)))
      (ess--command-proc-restore proc cmd-async-restore-alist))))

[lionel-]

Yes you must delete this after updating. Thank you.

[acid-bbq]

So far I haven't experienced the issue again. It looks it solved it. Thanks! Even though I've less used ESS these days because of vacation. But usually I should have occurred.

However, I have now a new issue. I still do the same thing as before, i.e.

1. I open an org file
2. I hit C-c ' to open a chunk of code. It opens it in a separate buffer
3. In this new buffer ESS[R], I send code with C-c C-n or C-c C-r or C-RET
4. I am asked for R starting project directory. I hit RET.
5. Now the R console opens in an iESS [R] buffer

After running some code and creating objects (e.g. vectors, data frames), from time to time, when I send code from the R script, that is the buffer opened in step 3 above, the code is sent with its result, but after, I have one random object I created before which is also always shown in the R console. It doesn't happen if I run code directly from the R console.

For example if I send this from the buffer:

2+2

It will display in the R console:

> 2+2 [1] 4 >[1] 0.1 0.2 0.3 0.4 >

where 0.1 0.2 0.3 0.4 is an object I previously created, e.g. x <- seq(0.1, 0.4, 0.1). Once it was a data frame, once a vector.

[lionel-]

Can you open a new issue please?