Regression: prompt with ANSI escape sequences

[klmr]

I am using an R prompt which includes ANSI escape sequences and the readline invisible character markers to have libreadline compute the correct prompt display width.

Unfortunately, the mere presence of these markers (e.g. without even ANSI escape sequence) is breaking Nvim-R; consider:

options(prompt = '\u01\u02>\u01\u02 ')

Having this in the ~/.Rprofile file and launching R inside Nvim via ,rf leads to the following error message, even when I configured g:Rout_prompt_str = "> ".

[Server] Divergent TCP message size: 0 x 67
Error detected while processing function ROnJobStdout:
line   40:
E115: Missing quote: '4.2.0^B.^B
Press ENTER or type command to continue
Error detected while processing function ROnJobStdout:
line   40:
E116: Invalid arguments for function SetNvimcomInfo
Press ENTER or type command to continue
[Server] TCP socket -1: restarting... Wrong TCP data length: 19 x 13

Subsequently, R launches but cannot be interacted with, and Nvim-R complains that “R is not ready yet” when running e.g. :Rhelp.

I was able to identify that commit 8a9293b261dab3f2d543ec060716bf8bcfb05066 introduced the breaking change by bisecting the log. Looking at the changes it seems that you are assigning special meaning to several characters, notably also \002. As far as I understand this also caused the issues in #784.

I don’t claim to understand what the code does but it seems that the purpose is to escape the string that is sent between R and Nvim-R. In this case, I think it would make sense to use proper, consistent escaping or encoding instead of reserving special characters, which will continue to run into problems. The easiest fix might be to base64-encode the string that’s transmitted, but I don’t know if there’s an easy way to base64 en-/decode strings in vimscript. A “proper” solution would implement string un-/escaping (i.e. transforming e.g. a\002"\nb" into a\\002\"\\nb before sending it, and reversing this after receiving) manually. This isn’t hard, but it does require iterating over the string and interpreting each character, which might be rather slow in vimscript.

[jalvesaq]

It's very fast to replace a byte with another. Other options, such as properly escaping special characters would require creating another buffer larger than the message being transmitted, and copying the bytes one at a time, replacing the special characters with escape sequences. But nvimcom must be very fast, otherwise, we would notice delays in R. Currently, nvimcom doesn't depend on any external library. If we adopted base64-encode, we would have to require users to install the dependencies. So, for now, I prefer to try possibly less problematic bytes. Currently, we are using:

• x01: used when sending TCP message from nvimcom to nvimrserver to indicate the end of the message.
• x02: used to indicate \n because \n causes the data to be considered finished in many circumstances, which fragments the communication.
• x04: used to indicate a single present in the contents of what is being transmitted from nvimcom to nvimrserver and from nvimrserver to nvim.
• x12: used to indicate single quotes in Vim/Lua dictionaries created by nvimcom.
• Some others that I think are mostly harmless in the context they are used.

Note that even x02 is harmless in the context of transmitting to nvim the current string used as the R prompt because nvim doesn't interpret the received data in this case. The problem is the coincidence between what I have chosen as "end-of-data" indicator and the special byte that you have to use (x01).

Looking at the ASCII table, I suspect that the less problematic bytes are the "device control" ones (x11, x12, x13, and x14).

[jalvesaq]

@klmr, could you please try the Nvim-R branch "bytes" and, if using cmp-nvim-r, also its "bytes" branch?

[klmr]

Thanks, that fixes the issue!

That being said, now I am getting back to the following behaviour, which displays the following otherwise benign message when starting R from within NeoVim:

Error detected while processing function ROnJobStdout[40]..SetNvimcomInfo[41]..Syntax Autocommands for "*"..function <SNR>19_SynSet[25]..script /home/rudolpk2/.local/share/nvim/lazy/Nvim-R/syntax/rout.vim:
line   93:
E401: Pattern delimiter not found: /^>^A^[[0m.*/
Press ENTER or type command to continue
Error detected while processing function ROnJobStdout[40]..SetNvimcomInfo[41]..Syntax Autocommands for "*"..function <SNR>19_SynSet[25]..script /home/rudolpk2/.local/share/nvim/lazy/Nvim-R/syntax/rout.vim:
line   93:
E475: Invalid argument: routInput /^>^A^[[0m.*/

My R prompt is

options(prompt = '\u01\u1b[34m\u02>\u01\u1b[0m\u02 ')

And I have let g:Rout_prompt_str = "> ".

So far I just accepted this as a minor nuisance but now it occurs to me that this might also be indirectly related (it seems to be a non-delimited regex due to the ANSI escape sequences). (If it’s unrelated and/or non-trivial to change, I am happy to open a new issue, but it’s also not serious and I am happy to live with it.)

[jalvesaq]

I can replicate the bug, but it should not happen (and, indeed, it doesn't happen on Vim, only on Neovim) because we have this in R/start_r.vim:

    if !exists('g:Rout_prompt_str')
        let g:Rout_prompt_str = substitute(Rinfo[2], ' $', '', '')
        let g:Rout_prompt_str = substitute(g:Rout_prompt_str, '.*#N#', '', '')
    endif

That is, the real R prompt is used only if Rout_prompt_str doesn't exist.

[jalvesaq]

Of course, the bug only affected Neovim: Vim does not highlight text in the terminal.

It's fixed now. I had replaced all \002 with \x14, but here it was written as \x02.

[klmr]

Jakson, as always your responsiveness is phenomenal! Thanks a lot.