Help with path for imports in python

[johnson-christopher]

I am trying to get YouCompleteMe to work with imports in a project that handles them in a different (probably bad, but I can't change it) way. The way it works is that the project's top level folder as well as one other sub folder are added as paths. So imagine this structure:

project_dir
├── tags
├── project_module
    ├── __init__.py
├── sub_dir
    ├── pythonfile.py
├── nested_project
    ├── lib
        ├── whatevermodule.py
    ├── nested_project_module
        ├── __init__.py

In pythonfile.py you can do the following:

import project_module
import nested_project_module
import lib.whatevermodule

And it finds all of those because the PYTHONPATH is updated to include project_dir and nested_project by the main program that's running all this. What I can't figure out is how to get YouCompleteMe to find these as well. I tried setting the working directory to the project_dir to handle the lib/mymodule.py one, but that did not work (using cd to change it manually after I figured out that it doesn't use Ctrl-Ps working dir, which would be a nice feature to add btw), plus there's no way it would have found the nested_project stuff. So I thought I'd found the answer when I learned that YouCompleteMe will use tag files. However, that isn't working either. Do tags files not handle this situation either, or is there something wrong with my YouCompleteMe setup that is preventing it from using it? I have the following in my .vimrc:

let g:ycm_filepath_completion_use_working_dir = 1
let g:ycm_collect_identifiers_from_tags_files = 1

And if I do 'echo tagfiles()' it does find the correct file. However any time I try to use any of these imported modules I just get a red status message saying 'Pattern not found.'

The tags file was generated using Exuberant Ctags with the --fields=+l and the -R options to get a single file for the entire project_dir with the language tag.

[vheon]

I'm just throwing my 2cent here. As my understanding when a semantic completion request is asked, YCM will forward the request to the semantic engine, so YCM in case that a semantic engine is supported is just a unifying layer between the user and the various engines. That being said, for python YCM uses jedi so I don't know how much this issue could be related to YCM directly. Maybe I just said a bunch of nonsense, only @Valloric could really tell.

[johnson-christopher]

I believe you are correct, and I've almost got it solved. If I set PYTHONPATH in my virtualenv's postactivate and then run vim from there it works great.

Now I just have to figure out a way to have vim update the PYTHONPATH it has and then reload YCM/Jedi whenever I switch to a given project. Manually updating PYTHONPATH from inside vim does not work, I assume because by that point YCM is already loaded using whatever it was when vim was started.

[johnson-christopher]

I'm marking this as closed as I do have a solution now that works for me. I'm not posting it because honestly it's ugly and I doubt anyone else needs it anyway. Basically I have an autocmd that calls a custom function any time I read, open or go back to a python buffer. The function checks if any of my project folders are in sys.path and if they are it removes them, then it checks if the current file is in a project folder and adds what it needs to sys.path for that project. If I can figure out a way to make it more generic as a python system path manager I may release that as its own vim plugin, but I doubt it since I don't think there'd be much demand and I've never written any vimscript.

[johnson-christopher]

On the off chance that anyone else has this issue I managed to solve it in a bit cleaner fashion by using python to do the actual updating. Here's how:

First, here is the function that does the update:

function! UpdatePythonPath()

    if !exists('g:current_python_path_mapping')
        let g:current_python_path_mapping = ''
    endif

python << EOF
import vim

import re
import sys

python_path_mappings = vim.eval('g:python_path_mappings')
path = vim.eval('expand("%:p")')
current_python_path_mapping = vim.eval('g:current_python_path_mapping')

for pattern, path_map in python_path_mappings.iteritems():
    r = re.compile(pattern)
    m = r.search(path)
    if m:
        name = path_map.get('name', '').format(**m.groupdict())
        if name != current_python_path_mapping:
            excludes = path_map.get('excludes', [])
            includes = path_map.get('includes', [])
            removes = [p for p in sys.path if any(re.match(x, p) for x in excludes)]
            adds = [i.format(**m.groupdict()) for i in path_map.get('includes', [])]
            sys.path = [p for p in sys.path if p not in removes] + adds
            vim.command('let g:current_python_path_mapping = "{0}"'.format(name))
        break
EOF

endfunction

Now in order to set up the mappings you need a variable called g:python_path_mappings which is a dictionary of dictionaries as follows:

let g:python_path_mappings = {
    \ 'path_to_match': {
        \ 'name': 'a_unique_name',
        \ 'excludes': ['list_of_paths_to_remove_from_sys_path'],
        \ 'includes': ['list_of_paths_to_add'],
        \ },
    \ }

And it supports Python regular expressions and groups, like this:

let g:python_path_mappings = {
    \ '(?P<full_project_path>.*/projects/[^/]+)/.*': {
        \ 'name': '{full_project_path}',
        \ 'excludes': ['(.*/)?projects/.*'],
        \ 'includes': ['{full_project_path}','{full_project_path}/ws'],
        \ },
    \ }

That says match anything in '/whatever/projects/some_project_name/', name it that path, exclude any other 'projects/some_other_project' folders, and then add both '/whatever/projects/some_project_name' and '/whatever/projects/some_project_name/ws'.

Then to have it called each time you open, read or go back to a Python file use this:

au BufEnter,BufNewFile,BufRead *.py :call UpdatePythonPath()

Or you can just look at the end of my .vimrc here: https://github.com/Raugturi/dotfiles/blob/master/.vimrc