search

ycm-core / YouCompleteMe

A code-completion engine for Vim
http://ycm-core.github.io/YouCompleteMe/
GNU General Public License v3.0
25.45k stars 2.81k forks source link

Latest versions use ycmd that tries (and fails) to use python3-isms with a python2 interpreter. #2024

Closed comand closed 8 years ago

comand commented 8 years ago

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside the brackets) before filing your issue:

Thank you for adhering to this process! It ensures your issue is resolved quickly and that neither your nor our time is needlessly wasted.

Issue Details

I updated to the latest YCM the other day, after successfully using previous versions for over a year now with the same setup. It appears that recent changes to ycmd for UTF support have brought along some Python3 syntax that doesn't work in Python2.

As an aside, I also noted that the latest version of the ycm_core support lib use clang-c/Documentation.h, which is not present in llvm 3.4. The readme indicates that YCM should work fine with an older version, but that's no longer the case.

Steps to reproduce

1) Install YCM (I used NeoBundle) 2) Build ycm_core for clang completion. 3) Open Vim/GVim on an empty .cpp file.

System Configuration

% cat /etc/issue Red Hat Enterprise Linux Workstation release 6.6 (Santiago)

Note that I'm using RHEL 6.6, which is fairly ancient, but I have a more recent set of tools, including gcc 4.8.2, Python 2.7.5, llvm 3.7.1, etc, that I compile vim and ycm_core against.

% vim --version VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Mar 2 2016 11:42:48) Included patches: 1-1468 Huge version without GUI. Features included (+) or not (-): +acl +farsi +mouse_netterm +tag_binary +arabic +file_in_path +mouse_sgr +tag_old_static +autocmd +find_in_path -mouse_sysmouse -tag_any_white -balloon_eval +float +mouse_urxvt -tcl -browse +folding +mouse_xterm +terminfo ++builtin_terms -footer +multi_byte +termresponse +byte_offset +fork() +multi_lang +textobjects +channel +gettext -mzscheme +title +cindent -hangul_input +netbeans_intg -toolbar -clientserver +iconv +packages +user_commands -clipboard +insert_expand +path_extra +vertsplit +cmdline_compl +job +perl +virtualedit +cmdline_hist +jumplist +persistent_undo +visual +cmdline_info +keymap +postscript +visualextra +comments +langmap +printer +viminfo +conceal +libcall +profile +vreplace +cryptv +linebreak +python/dyn +wildignore +cscope +lispindent -python3 +wildmenu +cursorbind +listcmds +quickfix +windows +cursorshape +localmap +reltime +writebackup +dialog_con +lua/dyn +rightleft -X11 +diff +menu +ruby/dyn -xfontset +digraphs +mksession +scrollbind -xim -dnd +modify_fname +signs -xsmp -ebcdic +mouse +smartindent -xterm_clipboard +emacs_tags -mouseshape +startuptime -xterm_save +eval +mouse_dec +statusline -xpm +ex_extra +mouse_gpm -sun_workshop
+extra_search -mouse_jsbterm +syntax
system vimrc file: "/etc/vimrc" user vimrc file: "$HOME/.vimrc" 2nd user vimrc file: "~/.vim/vimrc" user exrc file: "$HOME/.exrc" fall-back for $VIM: "/etc" f-b for $VIMRUNTIME: "/usr/share/vim/vim74" Compilation: /opt/tools/bin/gcc -c -I. -Iproto -DHAVE_CONFIG_H -O2 -g -pipe -Wall -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: /opt/tools/bin/gcc -L. -rdynamic -Wl,-export-dynamic -Wl,-E -L/usr/local/lib -Wl,--as-needed -o vim -lm -lelf -lnsl -lselinux -ltinfo -lacl -lattr -lgpm -ldl -Wl,-E -Wl,-R/opt/tools/lib -fstack-protector -L/usr/local/lib -L/opt/tools/lib/perl5/5.12.1/x86_64-linux/CORE -lperl -lnsl -ldl -lm -lcrypt -lutil -lc

YcmDebugInfo/YcmDebugToggleLogs

Initially, YCM doesn't even load, so these commands are unavailable. The vim messages console contains this:

Traceback (most recent call last): File "", line 19, in File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/setup.py", line 38, in SetUpYCM base.LoadJsonDefaultsIntoVim() File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/base.py", line 46, in LoadJsonDefaultsIntoVim defaults = user_options_store.DefaultOptions() File "/home/comand/.vim/bundle/YouCompleteMe/python/ycm/../../third_party/ycmd/ycmd/user_options_store.py", line 55, in DefaultOptions options = json.loads( ReadFile( settings_path ) ) File "/home/comand/.vim/bundle/YouCompleteMe/python/ycm/../../third_party/ycmd/ycmd/utils.py", line 54, in ReadFile with open( filepath, encoding = 'utf8' ) as f: TypeError: 'encoding' is an invalid keyword argument for this function

If I go into ycmd/utils.py and change the offending open call to use io.open instead, I get a bit further. When loading the same empty .cpp file, YCM crashes upon entering insert mode, with the following details:

:messages Messages maintainer: Bram Moolenaar Bram@vim.org "testme.cpp" 0L, 0C unsupported operand type(s) for ^: 'str' and 'str'

:YcmDebugInfo Printing YouCompleteMe debug information... Error detected while processing function 119_DebugInfo: line 2: Traceback (most recent call last): File "", line 1, in File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/youcompleteme.py", line 535, in DebugInfo 'debug_info' ) File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/client/base_request.py", line 70, in PostDataToHandler timeout ) ) File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/client/base_request.py", line 163, in JsonFromFuture _ValidateResponseObject( response ) File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/client/base_request.py", line 193, in _ValidateResponseObject if not SecureBytesEqual( our_hmac, their_hmac ): File "/home/comand/.vim/bundle/YouCompleteMe/python/ycm/../../third_party/ycmd/ycmd/hmac_utils.py", line 83, in SecureBytesEqual result |= x ^ y TypeError: unsupported operand type(s) for ^: 'str' and 'str' E858: Eval did not return a valid python object

:YcmToggleLogs stderr <log pane appears but it's empty>

Thanks in advance for any assistance you are able to offer.

[Is there a way for me to go back to an earlier version of the plugin and it's dependencies? I tried using the NeoBundle 'rev' argument to lock back to an earlier revision, but that doesn't appear to include the older revs of third_party submodules, so the plugin fails because ycmd is still the latest version, and the wrong support libraries are built (just ycm_core, no ycm_client_support).]

krak3n commented 8 years ago

I have the same problem on OSX where my default python install is 2.7.10 on OSX 10.11.13

comand commented 8 years ago

If I naively fix the issue inside of SecureBytesEqual (by catching TypeError and using x != y as a test instead), the next failure is here:

Traceback (most recent call last): Error detected while processing function youcompleteme#Complete: line 23: File "", line 1, in Error detected while processing function youcompleteme#Complete: line 23: File "/home/comand/.vim/bundle/YouCompleteMe/autoload/../python/ycm/base.py", line 58, in CompletionStartColumn Error detected while processing function youcompleteme#Complete: line 23: vimsupport.CurrentFiletypes()[ 0 ] ) - 1 ) Error detected while processing function youcompleteme#Complete: line 23: File "/home/comand/.vim/bundle/YouCompleteMe/python/ycm/../../third_party/ycmd/ycmd/request_wrap.py", line 109, in CompletionStartColumn Error detected while processing function youcompleteme#Complete: line 23: str( utf8_line_value[ : column_num -1 ], 'utf8' ) ) + 1 Error detected while processing function youcompleteme#Complete: line 23: TypeError: str() takes at most 1 argument (2 given) Error detected while processing function youcompleteme#Complete:

krak3n commented 8 years ago

I think everything in ycmd is python 3 with no support for python 2 so unless your default interpreter is python 3 I don't think its going to work?

comand commented 8 years ago

It looks like ycmd does try to be compatible with both 2 and 3 -- there are some uses of open, for instance, that pick the flags using "open(filePath, flags_for_py2 if PY2 else flags_for_py3)", and use of the python-future module, etc. I really do hope that support for Python 2 isn't being dropped, as it will likely be years before I'm able to use Python 3 in my work devel environment.

comand commented 8 years ago

It looks like many of these changes that appear to have broken Python 2 support came from here:

https://github.com/Valloric/ycmd/pull/358

comand commented 8 years ago

If I roll YouCompleteMe back to d534afb, then submodule update --recursive to get the older ycmd and associated dependencies, completion works.

micbou commented 8 years ago

This is probably an issue with one of your installed pip packages, using its own builtins module and conflicting with the one from python-future. See issue Valloric/ycmd#411. Since you are not the first to have this issue, we should maybe consider to import future.builtins instead of builtins, to avoid this kind of conflict.

The readme indicates that YCM should work fine with an older version, but that's no longer the case.

From the documentation:

NOTE: We STRONGLY recommend AGAINST use of the system libclang instead of the upstream compiled binaries. Random things may break. Save yourself the hassle and use the upstream pre-built libclang.

In other words, we only support the latest upstream binaries (3.7.1 currently, 3.8.0 soon).

comand commented 8 years ago

In other words, we only support the latest upstream binaries (3.7.1 currently, 3.8.0 soon).

Got it -- I wish I could use the upstream binaries, but I can't on a system this ancient. I'll soon be able to use RHEL 7.x, so hopefully the situation will improve somewhat.

There is this in the doc as well, though -- I was successfully using libclang 3.4.2 up until very recently

You can use the system libclang only if you are sure it is version 3.3 or higher, otherwise don't. Even if it is, we recommend using the official binaries from llvm.org if at all possible. Make sure you download the correct archive file for your OS.

Maybe that should read "You can use the system libclang only if you are sure it is version 3.7.1 or higher" -- that was the thrust of my statement in the comment you quoted.

In the mean time, I'll check out what's going on in the ycmd issue you mentioned and see whether I'm running into that specific thing.

Thanks for looking!

comand commented 8 years ago

Looks like this is related to Valloric/ycmd#411, as my Python has Pygments 2.0.2 installed, which includes a builtins.py.

vheon commented 8 years ago

Since you are not the first to have this issue, we should maybe consider to import future.builtins instead of builtins, to avoid this kind of conflict.

@micbou I think we should really consider doing it

comand commented 8 years ago

I take it back, it wasn't pygments, but pies2overrides that was supplying an unqualified 'builtins' module.

Valloric commented 8 years ago

@micbou I agree, we should use future.builtins instead of builtins (assuming they're the same).

krak3n commented 8 years ago

cleared out all my local python packages and all seems to be working again :+1:

puremourning commented 8 years ago

So I think there are 2 things being discussed here:

  1. use of future.builtins to avoid conflicts with other packages in the user's environment
  2. the addition of the GetDoc subcommand changed the minimum version of libclang with which the application works

I suspect 1. can and will be resolved, but 2. won't.

As we've made clear, we only support (and certainly only test) the version of libclang that is installed by install.py.

I sympathise with corporate users not able to use install.py directly , as I'm in the same situation. However, most users are able to build the tools from source (certainly, you can build clang 3.7 straight from SVN on RH 6.5 with the devtoolset-2 and use it with YCM).

I personally think the GetDoc feature is worth the upgrade. There are many other improvements in libclang since 3.4.

comand commented 8 years ago

Thanks for fixing!