ycm-core / YouCompleteMe

A code-completion engine for Vim
http://ycm-core.github.io/YouCompleteMe/
GNU General Public License v3.0
25.44k stars 2.81k forks source link

Null bytes in python docstrings prevents auto-completion from working properly #3207

Closed iswoqqe closed 5 years ago

iswoqqe commented 5 years ago

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside the brackets) before filing your issue:

Thank you for adhering to this process! It ensures your issue is resolved quickly and that neither your nor our time is needlessly wasted.

Issue Details

Python functions with docstrings containing null-characters will cause YouCompleteMe.GetCompletionResponse() (in the file YouCompleteMe/python/ycm/youcompleteme.py) to return an object that cannot be converted to a vim object. This prevents the auto-completion from working properly and causes the following errors:

Error detected while processing function <SNR>61_PollCompletion[8]..<SNR>61_Pyeval:
line    2:
E859: Failed to convert returned python object to vim value
Error detected while processing function <SNR>61_PollCompletion:
line    9:
E121: Undefined variable: completion_start_column
Error detected while processing function <SNR>61_PollCompletion:
line    9:
E15: Invalid expression: {   'start_column': response.completion_start_column,   'candidates': response.completions }

Possible explanation why objects cannot be converted

It seems like vim uses c-strings (which cannot contain null-characters) internally. I tried converting a couple of python objects to vim objects and got a return value of 0 and the error E859: Failed to convert returned python object to vim value for any object containing a string with a null-character.

pyeval gives the same result as py3eval for the above tests.

Steps to reproduce

Create two python files in the same directory:

# file: lib.py
def bad_function(x):
    """This docstring contains a null character \x00 <- here"""
    return True
# file: main.py
from . import lib
lib.[write part of function name here to get the error]

Possible solution

I believe it should be possible to simply remove any null bytes from the python strings before they are converted to vim strings without breaking anything.

Diagnostic data

Output of vim --version

VIM - Vi IMproved 8.1 (2018 May 18, compiled Oct 12 2018 19:49:42)
Included patches: 1-470
Compiled by Arch Linux
Huge version without GUI.  Features included (+) or not (-):
+acl               +extra_search      +mouse_netterm     +tag_old_static
+arabic            +farsi             +mouse_sgr         -tag_any_white
+autocmd           +file_in_path      -mouse_sysmouse    +tcl/dyn
+autochdir         +find_in_path      +mouse_urxvt       +termguicolors
-autoservername    +float             +mouse_xterm       +terminal
-balloon_eval      +folding           +multi_byte        +terminfo
+balloon_eval_term -footer            +multi_lang        +termresponse
-browse            +fork()            -mzscheme          +textobjects
++builtin_terms    +gettext           +netbeans_intg     +timers
+byte_offset       -hangul_input      +num64             +title
+channel           +iconv             +packages          -toolbar
+cindent           +insert_expand     +path_extra        +user_commands
-clientserver      +job               +perl/dyn          +vartabs
-clipboard         +jumplist          +persistent_undo   +vertsplit
+cmdline_compl     +keymap            +postscript        +virtualedit
+cmdline_hist      +lambda            +printer           +visual
+cmdline_info      +langmap           +profile           +visualextra
+comments          +libcall           +python/dyn        +viminfo
+conceal           +linebreak         +python3/dyn       +vreplace
+cryptv            +lispindent        +quickfix          +wildignore
+cscope            +listcmds          +reltime           +wildmenu
+cursorbind        +localmap          +rightleft         +windows
+cursorshape       +lua/dyn           +ruby/dyn          +writebackup
+dialog_con        +menu              +scrollbind        -X11
+diff              +mksession         +signs             -xfontset
+digraphs          +modify_fname      +smartindent       -xim
-dnd               +mouse             +startuptime       -xpm
-ebcdic            -mouseshape        +statusline        -xsmp
+emacs_tags        +mouse_dec         -sun_workshop      -xterm_clipboard
+eval              +mouse_gpm         +syntax            -xterm_save
+ex_extra          -mouse_jsbterm     +tag_binary        
   system vimrc file: "/etc/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H   -D_FORTIFY_SOURCE=2  -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       
Linking: gcc   -L. -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector -rdynamic -Wl,-export-dynamic -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.28/core_perl/CORE  -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -L/usr/local/lib -Wl,--as-needed -o vim        -lm -ltinfo -lelf -lnsl    -lacl -lattr -lgpm -ldl   -Wl,-E -Wl,-rpath,/usr/lib/perl5/5.28/core_perl/CORE -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fstack-protector-strong -L/usr/local/lib  -L/usr/lib/perl5/5.28/core_perl/CORE -lperl -lpthread -ldl -lm -lcrypt -lutil -lc   -L/usr/lib -ltclstub8.6 -ldl -lz -lpthread -lm     

Output of YcmDebugInfo

Printing YouCompleteMe debug information...
-- Client logfile: /tmp/ycm_93ail2gl.log
-- Server Python interpreter: /home/iswoqqe/.conda/envs/cleanpython/bin/python
-- Server Python version: 3.6.7
-- Server has Clang support compiled in: True
-- Clang version: clang version 7.0.0 (tags/RELEASE_700/final)
-- Extra configuration file found and loaded
-- Extra configuration path: /home/iswoqqe/git/dotfiles/.ycm_extra_conf.py
-- Python completer debug information:
--   Python interpreter: /home/iswoqqe/.conda/envs/cleanpython/bin/python
--   Python path: ['/home/iswoqqe/.conda/envs/cleanpython/lib/python36.zip', '/home/iswoqqe/.conda/envs/cleanpython/lib/python3.6', '/home/iswoqqe/.conda/envs/cleanpython/lib/python3.6/lib-dynload', /home/iswoqqe/.conda/envs/cleanpython/lib/python3.6/site-packages']
--   Python version: 3.6.7
--   Jedi version: 0.13.1
--   Parso version: 0.3.1
-- Server running at: http://127.0.0.1:57095
-- Server process ID: 12779
-- Server logfiles:
--   /tmp/ycmd_57095_stdout_5q17z3ny.log
--   /tmp/ycmd_57095_stderr_x1wd8dax.log

Contents of YCM, ycmd and completion engine logfiles

OS version, distribution, etc.

Manjaro linux - fully updated as of 2018-10-28 (using stable repositories)

Output of build/install commands

I do not belive this is relevant to the reported issue, please tell me if it is.

bstaletic commented 5 years ago

Thanks for the detailed report. Ideally this would get fixed in vim, because a workaround in YCM will impose a performance impact* on every completion request. However, from my brief testing, this seems to solve it.

diff --git a/python/ycm/youcompleteme.py b/python/ycm/youcompleteme.py
index d94c4f99..bb8cb61a 100644
--- a/python/ycm/youcompleteme.py
+++ b/python/ycm/youcompleteme.py
@@ -316,6 +316,10 @@ class YouCompleteMe( object ):
     response = self._latest_completion_request.Response()
     response[ 'completions' ] = base.AdjustCandidateInsertionText(
         response[ 'completions' ] )
+    # Vim can't pyeval strings with embedded \0 characters.
+    for completion in response[ 'completions' ]:
+      if 'info' in completion:
+        completion[ 'info' ] = completion[ 'info' ].replace( '\x00', '\\x00' )
     return response

@iswoqqe Can you try the patch and see if it works for you?

* I have not measured the performance impact, but, since we're looping over all completions, I wouldn't be surprised if this affects users of Windows.h and unlimited ycm_max_num_candidates.

iswoqqe commented 5 years ago

Removing null bytes from info seems to fix the issue for now. I'll leave another comment if i run into any problems.

micbou commented 5 years ago

This should be fixed by PR #3208.

@iswoqqe Is this docstring yours or from a third-party library? I am asking this because using null characters in a docstring is most likely a mistake (they won't be displayed when generating the documentation). If it's from a third-party library, you should tell its author to fix it.

bstaletic commented 5 years ago

they won't be displayed when generating the documentation

I did see the embedded NULL character and the docstring when I used YcmComplter GetDoc with the current master.

micbou commented 5 years ago

I did see the embedded NULL character and the docstring when I used YcmComplter GetDoc with the current master.

I am not talking about YCM but documentation tools like Sphinx. It's even worse when using the help function to print the docstring: in that case, the string is truncated at the first null character.

iswoqqe commented 5 years ago

@micbou The docstring was from https://github.com/libtcod/python-tcod. I'll tell them about the issue. Are you referring to pythons help function? Because it does not truncate the docstring on my machine.

micbou commented 5 years ago

Are you referring to pythons help function? Because it does not truncate the docstring on my machine.

Yes. Looks like it depends on the shell (or the terminal?): I found that it can be printed as ^@, can be ignored, can be interpreted as a new line, or the string is truncated.

bstaletic commented 5 years ago

Looks like it depends on the shell (or the terminal?)

Probably both, plus the $TERM variable and the lister invoked by python. My $MANPAGER is vim, so python's help invokes vim to display help.