ycmd infinite loop?

[kovidgoyal]

After a few minutes of editing this file: https://github.com/kovidgoyal/calibre/blob/master/src/calibre/gui2/tweak_book/file_list.py

ycmd.py pegs one CPU at 100% and leaks memory continuously at about 20 MB/s

Vanilla YCM works normally with that file. If there is some more debugging I can do, let me know.

vim 7.4.52 ycm installed as (linux, gentoo, amd64)

git clone https://github.com/Valloric/YouCompleteMe.git git checkout --track origin/ycmd git submodule update --init --recursive ./install.sh

[kovidgoyal]

Split off from #358

[Valloric]

I checked out the latest calibre sources from GitHub and randomly edited that file for a while using both semantic and identifier completion. RAM usage stays at a consistent 60 MB, no CPU pegged at 100%.

So I can't repro this.

What are your YCM settings in your vimrc?

[Valloric]

What I'm really curious about is have you configured YCM to read from tags files. #595 points to that possibly being a problem.

[kovidgoyal]

let g:ycm_collect_identifiers_from_comments_and_strings = 1 let g:ycm_seed_identifiers_with_syntax = 1 let g:ycm_complete_in_comments = 1 let g:ycm_collect_identifiers_from_comments_and_strings = 1 let g:ycm_autoclose_preview_window_after_completion = 1 let g:ycm_autoclose_preview_window_after_insertion = 1

The tags file I use while editing calibre is generated with ctags --python-kinds=-i -R --fields=+l src/calibre

and is 3.7MB

[Valloric]

Wait... how does your tags file get used by YCM when you don't have let g:ycm_collect_identifiers_from_tags_files = 1?

BTW you listed let g:ycm_collect_identifiers_from_comments_and_strings = 1 twice.

[kovidgoyal]

I didnt say my tags file is used by ycm, I just posted it since you said you were interested in it. I find completion using tags to be useless in large enough projects, like calibre.

As for the double entry, yeah that's a bug, but it should not affect anything.

[kovidgoyal]

Also note, that if you are trying to reproduce, the python path in my vim when editing calibre files contains the path to the src/calibre directory inside the calibre checkout. I dont know if ycmd.py clones the python path in vim or not, that may be worth investigating.

[kovidgoyal]

Here's a test case that reproduces my symptoms: http://files.calibre-ebook.com/betas/ycmd-test.tar.xz

extract it and edit the include file file_list.py like this

cd ycm && vim -u vimrc file_list.py

The tarball contains only pathogen and YouCompleteMe, alongwith my vimrc. Of course you will need ot recompile YCM for your computer before running it.

That test case is pretty clean, containing only pathogen and YCM, editing the included .py file causes ycmd to go nuts after about two or three edits.

[kovidgoyal]

Note that I am currently on a crappy3G connection, so the upload of ycmd-test.tar.xz will take about ten minutes.

[Valloric]

I dont know if ycmd.py clones the python path in vim or not, that may be worth investigating.

It doesn't.

Here's a test case that reproduces my symptoms: http://files.calibre-ebook.com/betas/ycmd-test.tar.xz

It's late here, I'll take a look at that tomorrow, thanks!

You may also want to try throwing pdb at ycmd and trying to get a stacktrace of where it's sitting in an inf loop. I'd put my money on a request hanging in EventNotification in handlers.py.

The best thing you could do is to put the following in your vimrc:

let g:ycm_server_use_vim_stdout = 1
let g:ycm_server_log_level = 'debug'

And then starting gvim (not vim). You'll see the server logs stream by in the console as you use Vim. Then get me the logs from after the hang.

[kovidgoyal]

I am trying to narrow down the problem, editing the following python snippet in the test environment I posted before:

import os
from PyQt4.Qt import QDialog

class Test(QDialog):

    def __init__(self, parent=None):
        QDialog.__init__(self, parent)
        self.

As you type self., completion triggers for the self identifier, but fails as soon as you type the period and ycmd.py starts using a CPU at 100%. The process completes after about 2 mins at which time ycmd is using 500MB of memory and completion works.

Interestingly, with vanilla YCM completion fails for self. and remains failed no matter how long you wait. However completion of os.p works. It may just be that ycmd does not abort jedi in the same way that vanilla YCM does and the rpoblem with file_list.py is in jedi?

[kovidgoyal]

I will try the debugging suggestions you posted the next time I have a little time.

[Valloric]

It may just be that ycmd does not abort jedi in the same way that vanilla YCM does and the rpoblem with file_list.py is in jedi?

I'm starting to think that the root cause of this problem is Jedi. Vanilla YCM uses Jedi 0.6, ycmd updated it to 0.7.

You could confirm this by going into the YouCompleteMe/third_party/jedi folder and checking out the 0.6 tag (the folder is a subrepo). Then try to repro the bug.

[kovidgoyal]

I tried replacing third_party/jedi with the folder from the vanilla YCM checkout, made no difference. Note that I have jedi installed in my PYTHONPATH globally as well (I use it for other things) I assume ycmd.py always loads its private copy?

[kovidgoyal]

If I run

python -c "import jedi; raw = open('test.py').read(); s = jedi.Script(raw, 8, 13, ''); completions = s.complete(); print (completions)"

I get an exception:

jedi.common.UncaughtAttributeError: 'PyQt4.QtCore.pyqtSignal' object has no attribute '__name__'

This is with my system jedi (0.7)

This should imply that completion should fail for self. That is what happens in vanilla YCM, however in ycmd.py there is a huge memory leak and after about 2 mins completion works (both with jedi 0.6 and 0.7)

[Valloric]

I can repro this with your new test file after installing python-qt on my Ubuntu machine. Looking into it.

[Valloric]

I can also repro this without YCM at all. Here's my run_jedi.py script:

#!/usr/bin/env python

import jedi

raw = open('test.py').read()
s = jedi.Script(raw, 8, 13, '')
completions = s.complete()

print (completions)

And test.py:

import os
from PyQt4.Qt import QDialog

class Test(QDialog):

    def __init__(self, parent=None):
        QDialog.__init__(self, parent)
        self.

Running run_jedi.py like this: PYTHONPATH=~/repos/YouCompleteMe/third_party/jedi:$PYTHONPATH ./run_jedi.py will repro this. I don't get an exception, I get a list of the completions. The process eats RAM until it hits about 500MB and then prints the completions.

I'm using Ubuntu Lucid 10.04 and have installed PyQt4 with sudo apt-get install python-qt.

If I check out the latest Jedi sources from upstream's dev branch, then I get your exception. So that's the easy fix; I'll just point YCM Jedi subrepo checkout to the latest Jedi sources.

Thanks for reporting this and helping me debug it!

[fweep]

I'm seeing this same symptom, and using @Valloric's test code above, I'm seeing that jedi is the problem for me as well. I'm not using Qt at all; I don't even have it installed. I can repro the memory usage problem by using the run_jedi.py and test.py scripts above, and simply replacing the Qt code with unittest and trying to complete self. on a class derived from unittest.TestCase.

Having done that, I know it's not YCM's fault, but as you seem to have resolved this issue, I'm curious if you can provide some insight into what the root cause may be? I'm running a current YCM (as of 22 Nov 2013) and completely rebuilt it, but my ycmd ends up taking up 3G of RAM and 150-190% CPU, and jedi appears to be the culprit.

[fweep]

I'll also note that I checked out the 0.6.0 tag in third_party/jedi and rebuilt YCM, and now it's working fine for me, so I think this change is what made my ycmd start eating tons of RAM. I noticed that recently ycmd was going crazy, and I just updated YouCompleteMe a few days ago. It seems like maybe jedi has a problem in 0.7.0.

[kovidgoyal]

I have removed the jedi repo from ycmd and installed jedi system wide at 35c05f3 alongwith my fix for this issue https://github.com/davidhalter/jedi/issues/331 and I have not noticed jedi going crazy for the last week or so.

[glittershark]

I was also seeing these symptoms in addition to no semantic completion in Python, and checked out Jedi 0.6.0 in the submodule and that fixed it.

[fweep]

This was fixed for me with the latest YCM, where @Valloric is using a recent dev branch version of jedi.

[fweep]

Either I spoke too soon, or something changed again. ycmd is back to near-constant >100% CPU usage and 3G footprint, and slowing Vim down so much that it can't even keep up with my typing. I'll try reverting back to a version from a couple months ago and see if that helps, because it was working fine for me back then. At this point I'm ready to give up on the context-aware completion and just fall back on SuperTab (or at least disable jedi in YCM if possible).

[Valloric]

@fweep I pulled in latest Jedi master, not latest Jedi dev. Does using Jedi dev fix this? I think there was some confusion about which Jedi branch had the fix.

[glittershark]

@Valloric it was the dev branch that fixed it for me

[glittershark]

Just updated YCM, 18dfe1d definitely fixes the issue

[Valloric]

Glad to hear it.

[fweep]

My most recent comment was a red herring. It turns out that my MacBook's battery died last week, and the whole machine was running at about 20% performance (surprisingly hard to diagnose with top/vmstat/etc.). I still think Jedi has some real performance issues in general (and I know that's not @Valloric's fault), but now that I'm on a working machine, it's back to being usable.