Omni-completion error for functions with complex formatting in documentation

[ashiklom]

Omni-completion (C-x C-o) works fine for functions with simple docstrings:

library(MASS)
mv| # <C-x C-o> here works fine; lists mvfft and mvrnorm, along with their documentation

However, for functions with more complex formatting in their documentation, completion fails:

library(dplyr)
sel|  # C-x C-o here throws the following errors...

Error detected while processing function CompleteR[86]..WaitRCompletion[1]..ROnJobStdout:
line    7:
E115: Missing quote: '\Sexpr[results=rd, stage=render]{lifecycle::badge("deprecated")} These functions now live in the tidyselect pack
age as ‘tidyselect::vars_select()’, ‘tidys
Error detected while processing function CompleteR[86]..WaitRCompletion[1]..ROnJobStdout:
line    7:
E116: Invalid arguments for function SetComplMenu
[ClientServer] Unknown command: elect::vars_rename()’ and ‘tidyselect::vars_pull()’.'}},{'word': 'select_vars', 'menu': 'func [dplyr]'
, 'user_data': {'cls': 'f' [...]

Another example, looking for dplyr::filter:

fil|. # C-x C-o

E115: Missing quote: '\Sexpr[results=rd, stage=render]{lifecycle::badge("deprecated")} dplyr used to offer twin versions of each verb
suffixed with an underscore. These versions had standard evaluation (SE) semantics: rather than taking arguments by code, like NSE ver
bs, they took arguments by value. Their purpose was to make it possible to program with dplyr. However, dplyr now uses tidy evaluation
 semantics. NSE verbs still capture their arguments, but you can now unquote parts of these arguments. This offers full programmabilit
y with NSE verbs. T
Error detected while processing function CompleteR[86]..WaitRCompletion[1]..ROnJobStdout:
line    7:
E116: Invalid arguments for function SetComplMenu
[ClientServer] Unknown command: hus, the underscored versions are now superfluous. Unquoting triggers immediate evaluation of its oper
and and inlines the result [...]
2 fewer lines; before #30  15 seconds ago
2 more lines; after #30  16 seconds ago

It looks like Nvim-R un-quotes the documentation prematurely and tries to interpret the rest of the documentation string as a vimscript function, which obviously fails.

My system information is below:

R version info: ``` R version 4.0.3 (2020-10-10) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Catalina 10.15.7 ``` Neovim version info: ``` NVIM v0.4.4 Build type: Release LuaJIT 2.0.5 Compilation: /usr/bin/clang -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedantic -Wno-unused-param eter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong - fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FRO M -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/build/config -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/src -I/usr/local/in clude -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/deps-build/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/ usr/include -I/usr/local/opt/gettext/include -I/tmp/neovim-20200808-89283-1us0ytm/neovim-0.4.4/build/src/nvim/auto -I/tmp/neovim-20200 808-89283-1us0ytm/neovim-0.4.4/build/include Compiled by brew@Catalina Features: +acl +iconv +tui See ":help feature-compile" system vimrc file: "$VIM/sysinit.vim" fall-back for $VIM: "/usr/local/Cellar/neovim/0.4.4/share/nvim" ``` Vimrc used in above example: ```vim filetype plugin indent on syntax enable call plug#begin('~/.local/share/vim-plug') Plug 'jalvesaq/Nvim-R', {'for': ['r', 'rmd']} call plug#end() let mapleader=" " let maplocalleader=" m" ```

[ashiklom]

~I just updated to Neovim v0.5 and the problem went away -- completion works exactly as it should, including for the dplyr functions above.~ I'm guessing the issue somewhere in here (https://github.com/jalvesaq/Nvim-R/blob/master/R/common_global.vim#L3794-L3817), perhaps because user_data must be a string?

~Anyway, upgrading Neovim solved the problem for me, but I'll leave this open because Neovim 0.5 is only in nightly release right now.~

EDIT: Actually, I am still hitting the same error -- it seemed to work at first, but now I'm hitting the same error again. Very strange. I tried deleting the .cache/Nvim-R directory, but that did not resolve the problem.

[jalvesaq]

I get the result below with both /usr/bin/nvim (0.4.4) and the current github version of nvim (0.5.0):

The \Sexpr code shouldn't be there but I don't get errors related to missing quotes.

[ashiklom]

Sorry for the long silence -- I haven't had a chance to dig into this until now.

I can confirm that I am not encountering this problem on a Linux HPC running R 3.5.3, only on my MacOS running R 4.0.4. (I haven't tried a Linux box with R 4.0.x yet).

After a lot of debug by fprintf, I've narrowed the problem down to something in the description string. If I modify the nclientserver.c file to use a dummy description string like this:

            p = str_cat(p, "', 'descr': '");
            /* p = str_cat(p, f[6]); */
            p = str_cat(p, "DESCRIPTION GOES HERE");
            p = str_cat(p, "'}},");

...and rebuild nvimcom, the completion works fine (albeit with the dummy description string for all functions).

I tried adding a fprintf(stderr, ...) statement in there to see what compl_buffer actually looks like at the end of this process and I noticed that the entry for select had this (full output in details below; emphasis mine):

except that no error is thrown for names that ***don''t*** exist

Note the double quotes in don''t.

However, I also get similar errors to the one reported in the original issue when I try to complete fil (with dplyr loaded), but I don't hit any errors when trying to complete filt. I also don't hit any errors when trying to complete fil without dplyr loaded. As far as I can tell, filter doesn't have any single quotes in its documentation.

Could this be an OS/compiler-specific variation in how C works with character strings? Perhaps it freaks out when the list of completion candidates gets too long, or perhaps I'm running up against the upper limit of the allocated completion buffer size in a weird way? Those are my best guesses.

---

``` { 'word': 'select.list', 'menu': 'func [utils]', 'user_data': { 'cls': 'f', 'pkg': 'utils', 'usage': [['choices'], ['preselect', 'NULL'], ['multiple', 'FALSE'], ['title', 'NULL'], ['graphics', 'getOption("menu.graphics")'], ], 'ttl': 'Select Items from a List', 'descr': 'Select item(s) from a character vector.', } }, { 'word': 'selfStart', 'menu': 'func [stats]', 'user_data': { 'cls': 'f', 'pkg': 'stats', 'usage': [['model'], ['initial'], ['parameters'], ['template'], ], 'ttl': 'Construct Self-starting Nonlinear Models', 'descr': 'Construct self-starting nonlinear models to be used in ‘nls’, etc. Via function ‘initial’ to compute approximate parameter values from data, such models are “self-starting”, i.e., do not need a ‘start’ argument in, e.g., ‘nls()’.', } }, { 'word': 'selectMethod', 'menu': 'func [methods]', 'user_data': { 'cls': 'f', 'pkg': 'methods', 'usage': [['f'], ['signature'], ['optional', 'FALSE'], ['useInherited', 'TRUE'], ['mlist', 'if (!is.null(fdef)) getMethodsForDispatch(fdef)'], ['fdef', 'getGeneric(f, !optional)'], ['verbose', 'FALSE'], ['doCache', 'FALSE'], ], 'ttl': 'Get or Test for the Definition of a Method', 'descr': 'The function ‘selectMethod()’ returns the method that would be selected for a call to function ‘f’ if the arguments had classes as specified by ‘signature’. Failing to find a method is an error, unless argument ‘optional = TRUE’, in which case ‘NULL’ is returned. The function ‘findMethod()’ returns a list of environments that contain a method for the specified function and signature; by default, these are a subset of the packages in the current search list. See section “Using ‘findMethod()”’ for details. The function ‘getMethod()’ returns the method corresponding to the function and signature supplied similarly to ‘selectMethod’, but without using inheritance or group generics. The functions ‘hasMethod()’ and ‘existsMethod()’ test whether ‘selectMethod()’ or ‘getMethod()’, respectively, finds a matching method.', } }, { 'word': 'selectSuperClasses', 'menu': 'func [methods]', 'user_data': { 'cls': 'f', 'pkg': 'methods', 'usage': [['Class'], ['dropVirtual', 'FALSE'], ['namesOnly', 'TRUE'], ['directOnly', 'TRUE'], ['simpleOnly'], ['where', 'topenv(parent.frame())'], ], 'tl': 'Super Classes (of Specific Kinds) of a Class', 'descr': 'Return superclasses of ‘ClassDef’, possibly only non-virtual or direct or simple ones. These functions are designed to be fast, and consequently only work with the ‘contains’ slot of the corresponding class definitions.', } }, { 'word': 'select', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['.data'], ['...'], ], 'ttl': 'Subset columns using their names and types', 'descr': 'Select (and optionally rename) variables in a data frame, using a concise mini-language that makes it easy to refer to variables based on their name (e.g. ‘a:f’ selects all columns from ‘a’ on the left to ‘f’ on the right). You can also use predicate functions like is.numeric to select variables based on their properties. \subsection{Overview of selection features}{ Tidyverse selections implement a dialect of R where operators make it easy to select variables: • ‘:’ for selecting a range of consecutive variables. • ‘!’ for taking the complement of a set of variables. • ‘&’ and ‘|’ for selecting the intersection or the union of two sets of variables. • ‘c()’ for combining selections. In addition, you can use \strong{selection helpers}. Some helpers select specific columns: • ‘everything()’: Matches all variables. • ‘last_col()’: Select last variable, possibly with an offset. These helpers select variables by matching patterns in their names: • ‘starts_with()’: Starts with a prefix. • ‘ends_with()’: Ends with a suffix. • ‘contains()’: Contains a literal string. • ‘matches()’: Matches a regular expression. • ‘num_range()’: Matches a numerical range like x01, x02, x03. These helpers select variables from a character vector: • ‘all_of()’: Matches variable names in a character vector. All names must be present, otherwise an out-of-bounds error is thrown. • ‘any_of()’: Same as ‘all_of()’, except that no error is thrown for names that don''t exist. This helper selects variables with a function: • ‘where()’: Applies a function to all variables and selects those for which the function returns ‘TRUE’. }', }, }, { 'word': 'select_', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['.data'], ['...'], ['.dots', 'list()'], ], 'ttl': 'Deprecated SE versions of main verbs.', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("deprecated")} dplyr used to offer twin versions of each verb suffixed with an underscore. These versions had standard evaluation (SE) semantics: rather than taking arguments by code, like NSE verbs, they took arguments by value. Their purpose was to make it possible to program with dplyr. However, dplyr now uses tidy evaluation semantics. NSE verbs still capture their arguments, but you can now unquote parts of these arguments. This offers full programmability with NSE verbs. Thus, the underscored versions are now superfluous. Unquoting triggers immediate evaluation of its operand and inlines the result within the captured expression. This result can be a value or an expression to be evaluated later with the rest of the argument. See ‘vignette("programming")’ for more information.', }, }, { 'word': 'select_all', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['.tbl'], ['.funs', 'list()'], ['...'], ], 'ttl': 'Select and rename a selection of variables', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("superseded")} ‘rename_if()’, ‘rename_at()’, and ‘rename_all()’ have been superseded by ‘rename_with()’. The matching select statements have been superseded by the combination of a ‘select()’ + ‘rename_with()’. These functions were superseded because ‘mutate_if()’ and friends were superseded by ‘across()’. ‘select_if()’ and ‘rename_if()’ already use tidy selection so they can''t be replaced by ‘across()’ and instead we need a new function.', }, }, { 'word': 'select_at', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['.tbl'], ['.vars'], ['.funs', 'list()'], ['...'], ], 'ttl': 'Select and rename a selection of variables', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("superseded")} ‘rename_if()’, ‘rename_at()’, and ‘rename_all()’ have been superseded by ‘rename_with()’. The matching select statements have been superseded by the combination of a ‘select()’ + ‘rename_with()’. These functions were superseded because ‘mutate_if()’ and friends were superseded by ‘across()’. ‘select_if()’ and ‘rename_if()’ already use tidy selection so they can''t be replaced by ‘across()’ and instead we need a new function.', }, }, { 'word': 'select_if', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['.tbl'], ['.predicate'], ['.funs', 'list()'], ['...'], ], 'ttl': 'Select and rename a selection of variables', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("superseded")} ‘rename_if()’, ‘rename_at()’, and ‘rename_all()’ have been superseded by ‘rename_with()’. The matching select statements have been superseded by the combination of a ‘select()’ + ‘rename_with()’. These functions were superseded because ‘mutate_if()’ and friends were superseded by ‘across()’. ‘select_if()’ and ‘rename_if()’ already use tidy selection so they can''t be replaced by ‘across()’ and instead we need a new function.', }, }, { 'word': 'select_var', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['vars'], ['var', '-1'], ], 'ttl': 'Select variables', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("deprecated")} These functions now live in the tidyselect package as ‘tidyselect::vars_select()’, ‘tidyselect::vars_rename()’ and ‘tidyselect::vars_pull()’.', } }, { 'word': 'select_vars', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['vars', 'chr()'], ['...'], ['include', 'chr()'], ['exclude', 'chr()'], ], 'ttl': 'Select variables', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("deprecated")} These functions now live in the tidyselect package as ‘tidyselect::vars_select()’, ‘tidyselect::vars_rename()’ and ‘tidyselect::vars_pull()’.', } }, { 'word': 'select_vars_', 'menu': 'func [dplyr]', 'user_data': { 'cls': 'f', 'pkg': 'dplyr', 'usage': [['vars'], ['args'], ['include', 'chr()'], ['exclude', 'chr()'], ], 'ttl': 'Deprecated SE versions of main verbs.', 'descr': '\Sexpr[results=rd, stage=render]{lifecycle::badge("deprecated")} dplyr used to offer twin versions of each verb suffixed with an underscore. These versions had standard evaluation (SE) semantics: rather than taking arguments by code, like NSE verbs, they took arguments by value. Their purpose was to make it possible to program with dplyr. However, dplyr now uses tidy evaluation semantics. NSE verbs still capture their arguments, but you can now unquote parts of these arguments. This offers full programmability with NSE verbs. Thus, the underscored versions are now superfluous. Unquoting triggers immediate evaluation of its operand and inlines the result within the captured expression. This result can be a value or an expression to be evaluated later with the rest of the argument. See ‘vignette("programming")’ for more information.' } } ```

[jalvesaq]

I still get the same result shown in the screenshot that I posted on December 4, 2020. It's obviously wrongly formatted because the LaTeX like syntax isn't parsed, but there are no errors in the Vim script. Indeed, the single quotes are duplicated to avoid errors in the Vim script.

[ashiklom]

That's good to know, thanks! I'll try comparing the omnicompletion lists generated on my MacOS with those generated on a similarly configured Linux machine to see if there are any differences.

[jalvesaq]

@ashiklom, do you still see the bug in your Mac OS? Single quotes are now saved in the omnls_ files as "\004" and the byte "\004" is replaced with single quotes only during the float window creation.

[ashiklom]

Thanks! Unfortunately, I'm still seeing the error on MacOS. When trying to autocomplete the string fil (with dplyr loaded), I get:

Error detected while processing function CompleteR[94]..WaitRCompletion[1]..ROnJobStdout:
line    7:
E115: Missing quote: 'func [sta
Error detected while processing function CompleteR[94]..WaitRCompletion[1]..ROnJobStdout:
line    7:
E116: Invalid arguments for function SetComplMenu
[ClientServer] Unknown command: ts]', 'user_data': {'cls': 'f', 'pkg': 'stats', 'usage': [['x'], ['filter'], ['method', 'c("convolution", "recursive")'], ['side [...]

Based on the latter, I'm not sure this is a quoting issue. Rather, could this have to do with some kind of unique buffer flushing behavior in MacOS / BSD?

I'm sure this is a frustrating issue for you to debug without more information from me, and I'm sorry I haven't found the time to investigate further! Feel free to close this issue as wontfix in the meantime until I can find the time to get a better sense of what's causing this.

[jalvesaq]

Maybe it's fixed now in the branch omni_tmp. Could you try it, please?

[ashiklom]

Yes, that fixed it! Completion on fil<|> now works beautifully and very efficiently. Thanks so much!

[jalvesaq]

Thanks for the feedback! I'll let the branch active for a few days before merging it. Please, report any new bug that you find.