Using YouCompleteMe as (better) python 2/3 completer with virtualenv

[reinhrst]

Using YouCompleteMe as python completer

Recently I’m working on a large project in Python (3). Vim is the editor of my choice, and so far it has proved a challenge to get a good completer in place. I love YouCompleteMe, however it still lacks some features that would make it well suited for python. The goal of this issue is to list the problems I see, to suggest ways to improve them, and possibly get feedback on these plans. I intend to tackle (some of) them in the near future, however it will be something that will be mostly weekend/night work, so progress may be slow.

Python 3 support

As far as I understand it’s important to distinguish three different parts here:

YouCompleteMe doesn’t run if vim is compiled with python 3

This is a problem, since vim only supports either python2 or python3. I currently have no need to compile vim with python3, however one can imagine that in the future there will be (other, desirable) vim plugins that need python3. I can see myself in the future writing a vim plugin that interfaces with the code I use on my project, which means it should run python3.

I think at some point YouCompleteMe should take another step towards maturity, by making this part of the code run in both python2 and python3. This is far from trivial, however if this code has good test coverage (I didn’t look at that), it may be nothing more than just a lot of work. At this point, this task is way down on my priority list.

YouCompleteMe Server (ycmd) doesn’t run on python 3

As far as I understand, there is (at most) 1 YouCompleteMe server running for all your vim instances. This server runs on a standalone python (2), and does not use vim’s python. This means that it is not a problem at all that it only runs on python 2 (or python 3 for that matter, or Go, or Julia, or Ruby). It’s just a dependency to the YouCompleteMe Server that could, worst case, in the future, be packaged with ycmd. The only problem with ycmd running on python 2 is the next point, which should be solved in some other way.

YouCompleteMe cannot complete python3 code

YouCompleteMe uses jedi as backend to handle some of the intelligence on what completions to offer for python. Jedi can handle both python 2 and python 3 code, however the parser only understands code for the python version it’s currently running at. At the moment, jedi is tightly bound to ycmd, i.e. ycmd just does an import jedi and calls the correct functions. As a result, jedi will always run in python 2 (since ycmd can only run on python 2).

It should be mentioned that jedi on python 2 does quite a decent job in offering completions for python 3, but it still is suboptimal, and I imagine things will only get worse with python 3.5 implementing pep-492, which adds new syntax.

Previously it was suggested that the best way around this, was to split off the jedi code into a separate server, that could be started with whatever version of python one needs. This indeed seems to be the best idea. One can even imagine having multiple jedi servers running at the same time, for different files in vim needing different python versions (how YouCompleteMe knows which file wants which version is something I’m getting to later). Although some precautions will have to be taken so that orphaned servers die, and that servers are not vulnerable to CSRF attacks, the amount of work needed to implement this, seems not too much.

YouCompleteMe cannot deal with virtualenv

Where a couple of years back, most people seemed content to install all their packages globally, by now using virtualenvs is the norm. YouCompleteMe is ignorant of virtualenvs, which means that it (=jedi) will use whatever virtualenv is active at the time the server starts. This will be an obvious problem if one is editing multiple files that not all reside in the same virtualenv.

Using the jedi-server solution described above, a separate jedi server could be started per virtualenv. Jedi doesn’t know about virtualenvs either, but it will serve completion-suggestions for whatever virtualenv it is started in.

Since a virtualenv defines a version of python in addition to installed packages, it seems to a very nice idea just run the jedi-server in a virtualenv and let the virtualenv deal with the rest.

The one problem we run into here, is how to know which virtualenv to use for which file. I see several options, and I would love some feedback on which one to choose: Use whatever virtualenv vim is using. Downside is that this means at any point only 1 virtualenv can be active (and if editing 2 files from different virtualenvs, one has to constantly switch). Nice thing is that it works “out of the box”, if a user first starts the virtualenv and then starts vim. Put this information in a YouCompleteMe config file, like the .ycm_extra_conf.py file. I think that currently this file is only being used for c/c++ languages but this can possibly be extended. If we use this file, it will result in a big ugly (but very necessary) warning when starting vim, on whether we trust this file enough to run it. Use a virtualenv name defined in a .venv file. A .venv file is supported to auto-activate a virtualenv in virtualenvwrapper or virtualfish. Disadvantage is that just naming a virtualenv (instead of using a path to the virtualenv) supposes that virtualenvwrapper (or similar) is being used. An alternative could be to have a link to the virtualenv path in the .venv file. I can see that, especially when using a path, this may also require some big fat warning at the start. For now I opt for the first option, just because it’s the fastest to implement. However I think that eventually one of the other 2 options would be better. (Note that in this case, one has to set g:ycm_path_to_python_interpreter to point to python 2 manually. Probably that is a good idea anyways if sometimes you may be starting a virtualenv before you start vim.

YouCompleteMe doesn’t like string-indices to dictionaries

I constantly find myself typing jsonobject[“params”][“foobar”] in my code, and params and foobar are not completed. This behaviour can be controlled using g:ycm_collect_identifiers_from_comments_and_strings. In C-like languages, there is probably there is little reason to ever collect identifiers from strings (let alone comments) and I’m sure that this option was added to YouCompleteMe mostly has a helpful hand to other languages. I however don’t want my pydoc strings collected, my user help messages collected, and definitely not my comments. I think python completion would greatly benefit from a smarter collection strategy here. Not entirely sure yet what it should be, but if it collected only all single-worded strings, or strings that are surrounded by []’s, that would make my life a lot happier. Shouldn’t be too hard, just calls for a slightly more fine-grained config option to do this.

Todo

As mentioned, I would be happy taking on some of these tasks in my weekends, beginning with making jedi a server. I would love to hear comments on things that I misunderstood or misrepresented here, or other suggestions you might have. Possibly I should discuss all this on another forum, or I should be pointed towards someone with similar plans. I’m open for anything :).

[reinhrst]

And just to make sure this post has the right tone: although I describe YouCompleteMe doesn't do this and that, I mainly mean it does do 99% of the things very well, and I just describe the 1% that's left :)

[oblitum]

fyi, if you care, I have a good hybrid YCM/neocomplete setup in my dotfiles:

• https://github.com/oblitum/dotfiles/blob/osx/.vimrc
• https://github.com/oblitum/dotfiles/blob/linux/.vimrc
• https://github.com/oblitum/dotfiles/blob/osx/.aliases
• https://github.com/oblitum/dotfiles/blob/linux/.aliases

It supports Vim compiled with python 2 and python 3 (my Arch works with that) as well as vim compiled with python 2 and a separate instance compiled with python 3 (I use this scheme on OS X and Debian-based distributions).

So, when I need to develop for python 3, I start the vim instance compiled with python 3, which will use neocomplete. In Arch I don't need to care to compile two Vim's since it works with py2 & py3, so when I start vim inside some virtualenv, it'll work with the python instance of that env and use YCM or neocomplete accordingly.

The .vimrc is setup with a plugin for pyenv. I use this setup in my working routine, no problems so far.

[vheon]

As far as I understand, there is (at most) 1 YouCompleteMe server running for all your vim instances

No, there is one instance of ycmd (the YouCompleteMe server) for every instance of vim.

YouCompleteMe cannot complete python3 code

... Previously it was suggested that the best way around this, was to split off the jedi code into a separate server, that could be started with whatever version of python one needs. This indeed seems to be the best idea. One can even imagine having multiple jedi servers running at the same time, for different files in vim needing different python versions (how YouCompleteMe knows which file wants which version is something I’m getting to later). Although some precautions will have to be taken so that orphaned servers die, and that servers are not vulnerable to CSRF attacks, the amount of work needed to implement this, seems not too much.

Actually I started implementing this as you can see in https://github.com/Valloric/ycmd/pull/160. I'm not a python programmer at all so I don't know if what I wrote is some good python code. Plus recently I got a new job, so before I can continue working on this I have to clear it with the management. But again if you are a python programmer and think that what I did is not optimal, please let me know what I did wrong or if you want to implement this yourself :+1:

Since a virtualenv defines a version of python in addition to installed packages, it seems to a very nice idea just run the jedi-server in a virtualenv and let the virtualenv deal with the rest.

I don't like the idea of depending on some third party tool. The way I see it is that if I activate a virtualenv, then I would restart the JediHTTP server with :YcmCompleter RestartServer and that would mean that jedi now can pick up whatever I need.

[reinhrst]

@oblitum thanks for your workaround. As far as I can tell, neocomplete is the lesser cousin to YouCompleteMe so I prefer YouCompleteMe to have good support for this. But in the meantime this may work very well.

@vheon This is exactly the kind of answer I was hoping for. I had been looking for any progress on this, but hadn't found anything.

I gave the virtualenv selection mechanism some more thought, and I think you're right for now in just choosing the one that is active in vim., especially since you clarified that I can start multiple vims, and vim will have its own ycmd. I could imagine a situation where I would be working in the same vim on a server and client part of the same program, wanting to be able to have each run in its own virtualenv, but let's solve one problem at a time.

I will look at the pull request. Again many thanks! For now closing this, will re-create something when I have time to look at the string-indices thing!

[vheon]

I could imagine a situation where I would be working in the same vim on a server and client part of the same program, wanting to be able to have each run in its own virtualenv, but let's solve one problem at a time.

I have some ideas for that as well, but as you said, one problem at a time :+1: