search

ycm-core / YouCompleteMe

A code-completion engine for Vim
http://ycm-core.github.io/YouCompleteMe/
GNU General Public License v3.0
25.44k stars 2.81k forks source link

numpy, timeout and YCM dying. #2192

Closed ocehugo closed 7 years ago

ocehugo commented 8 years ago

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside the brackets) before filing your issue:

Thank you for adhering to this process! It ensures your issue is resolved quickly and that neither your nor our time is needlessly wasted.

Issue Details

vim version : https://gist.github.com/5cb3c50da169bcb857ae8f6d9e93faf1

YcmDebugInfo: Printing YouCompleteMe debug information... -- Server has Clang support compiled in: False -- JediHTTP running at 127.0.0.1:60053 -- python binary: /usr/local/bin/python3 -- stdout log: /var/folders/7x/0q8hpfd161bf5zpc0kk8fcxh0000gn/T/ycm_temp/jedihttp_6005 3_stdout.log -- stderr log: /var/folders/7x/0q8hpfd161bf5zpc0kk8fcxh0000gn/T/ycm_temp/jedihttp_6005 3_stderr.log -- Server running at: http://127.0.0.1:60045 -- Server process ID: 18900 -- Server logfiles: -- /var/folders/7x/0q8hpfd161bf5zpc0kk8fcxh0000gn/T/ycm_temp/server_60045_stdout.log -- /var/folders/7x/0q8hpfd161bf5zpc0kk8fcxh0000gn/T/ycm_temp/server_60045_stderr.log stderr: https://gist.github.com/403a92de05d986f33573d060a0e7bf8d OS: OSX 15.5.0

Description: YCM is dying just after a timeout is raised. I would expect that after a timeout YCM will just ignore and continue running for the next completion.

I can reproduce this quite easy when using numpy, and happens if the user is quick. For me this happens all the time ( i can't use ycm anyomre with python since the timeout msg keep happening and i need to restart vim). This is raised if the user do not wait for the completion box to appears (i.e. is quicker than the default 0.5 seconds timeout and keep typing) . This is particular worse with the numpy python package since it's a big one. Decreasing YCM timeout, for example for 0.1 , generate the same problem (YCM and python crashes) but later (not soo later, if the user keep typing other classes in a row, it will happen after the 3 or so 'class typing'.

How to reproduce: Open a file and type the following code as quick as possible ( particular the variable assignment).

import numpy as np
a = np.array(1)
b = np.array(2)
c = np.array(3)
d = np.array(4)

Apart from the timeout msg (which is quite annoying since it will create missing characters when trying to typing the code), the Ycm will crash and a python unexpected termination will show up in the screen (for osx). After that when trying to complete or navigate in the classname space a Connection error will start to show:

ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=60358): Max retries exceeded
with url: /completions (Caused by NewConnectionError('<requests.packages.urllib3.connect
ion.HTTPConnection object at 0x110d12ed0>: Failed to establish a new connection: [Errno
61] Connection refused',))

Other completions like ultisnips still works, but all completions related to numpy generate the error above or timeouts msgs. I will keep like that for the session, making editing the file pratically impossible and requiring restarting vim. Ycmrestartserver do not change this.

So there is 2 different problems:

  1. YCM crash
  2. Timeout and Connection Error keep disrupting the work environment after the crash.

I would expect that:

  1. YCM do not crash after timeouts
  2. If crash, YCM will not interrupt vim anymore.

PS: I believe there is something related to retries/timeouts and etc but I believe, at last from this experience, this retries or timeouts maybe not applicable for some python packages. PS2: I believe that if YCM has an option YCM_IGNORE_ALL_MSGS will be good at least to avoid interrupting the typing process.

Valloric commented 8 years ago

@vheon This seems related to Jedi since I see numpy & Python.

vheon commented 8 years ago

I'll try to debug it this weekend 👍

bijancn commented 8 years ago

When I start writing the test case, I get after the first np. the error HTTPConnectionPool(host='127.0.0.1', port=56020): Read timed out. (read timeout=0.5) but no further error output. If I wait a tiny bit numpy seems to get loaded and semantic completion works.

Vim version 7.4 patches 1-1689 and +python3 -python. ycmd without clang support. YouCompleteMe on 7f21fb6.

gauteh commented 8 years ago

777: I think the timeout should be configurable.

vheon commented 8 years ago

When I start writing the test case, I get after the first np. the error HTTPConnectionPool(host='127.0.0.1', port=56020): Read timed out. (read timeout=0.5) but no further error output. If I wait a tiny bit numpy seems to get loaded and semantic completion works.

@bijancn that is jedi which on the first attempt of completion has to parse a lot of python files and could not give us back the completion before the timeout but after the first attempt usually you should not hit the timeout. As a side note, I remember that numpy is a big library and jedi had some problem working with it.

@gauteh I will quote the response made on #742

There is no option to change the timeout value because you should never need it to be higher than 0.5 seconds. If that's not high enough, you have a different problem you need to fix.

vheon commented 8 years ago

@vheon This seems related to Jedi since I see numpy & Python.

I gave this a quick look and the problem seems to be that we cannot give the user what he ask for in time. In fact, I tried wrote a file like:

import numpy as np
a = np.

and I reached the timeout but if I exit insert mode, re-enter it and press <C-Space> to force the completion I get the completions. What surprised me was that even thought we hit the timeout YCM didn't fallback to the Identifier completer 😕

ocehugo commented 8 years ago

@vheon you can get the completions if you do this but if you continue:

import numpy as np
a = np.array([0])

and after

b = np.zeros([0])

without waiting for the completion to happen, YCM will crash and start to raise HTTPconnectionpool errors each time you are navigating after the "." or writing newlines with np methods.

qiuqiufrank commented 8 years ago

@ocehugo I have same problem,did you fix it yet? please!!!

dojoteef commented 7 years ago

@vheon I am running into the same issue. I have a suggestion for a fix, but have not had a chance to look look at the current code to check for feasibility of implementing the feature. I have only thought through it in the case of Python, but I'm sure it's applicable to other languages.

It basically just comes down to caching. Specifically allowing the cache to be saved to disk to be reused. Additionally do not wait until a user tries to access a completion before trying to process the imports, instead begin caching the data immediately upon opening an existing file.

Most IDEs do something similar, so it's not a stretch to assume something like this could be useful for larger projects or for importing large libraries.

The idea would be to have a cache that you check first. It has a SHA or some similar hash to determine if the imported code has been modified. If so, then reparse the library/code that is being imported otherwise just use the cache. The first time a new file is seen it begins trying to cache anything need to get the completions.

If the concern is that users of the plugin might not expect the behavior that lots of new files will be cached onto their computer just for using the completion feature it can be disabled by default. Additionally it would be good to have a separate cache for system level libraries versus local ones, such that the system level libraries would not need to be reprocessed on a regular basis, thus making it very fast in the general case.

I'd be happy to open this as a new issue/feature request that can be tracked separately if this is something that you believe is within scope of the project.

vheon commented 7 years ago

@dojoteef I think that, very slowly, we're working towards a solution like the one you suggested (and by "we" I think I'm referring to @micbou here 😜 ). The cache on disk is already implemented by jedi itself, in fact only the first call to jedi is usually slow because it has to parse the code for the first time, but it cache the results on disk so the next call to jedi is actually much faster. The other thing that I suggested on the ycm-users mailing list which is to preload the imports that the user has on the buffer. @micbou actually implemented a couple of API on JediHTTP that will make this possible; specifically I'm taling about https://github.com/vheon/JediHTTP/pull/23 and https://github.com/vheon/JediHTTP/pull/24. So I believe we only need to implement the bits in ycmd and then we can make things better in this regard.

dojoteef commented 7 years ago

@vheon Thanks for the quick reply! It's nice to know that you all already have a handle on an approach for the issue. BTW, I really appreciate all the hard work everyone who contributes to YouCompleteMe is doing.