hasufell commented 5 years ago

Problems summary

I can't provide a minimal config, nor a minimal test case, because there is no minimal test case for this problem.

Working on a project with ~4k LOC python code over a longer period of time eventually consumes so much memory on my machine, that the OOM killer gets triggered for other programs. It seems there are a lot of deoplete processes spawned and they accumulate memory over time. I have to kill them manually.

As a comparison I used YCM, which also has autocompletion, but doesn't have deopletes memory behavior.

The problem now is, this doesn't become visible on trivial projects and is more a performance/process/"memory-leak" issue. Any suggestions on how I can debug this further once it happens again would be appreciated.

Environment Information

deoplete version (SHA1): a711749ad6f6a455f64c0677f1a7c70b9550c6ca
OS: Linux
neovim/Vim :version output:

$ vim --version
VIM - Vi IMproved 8.1 (2018 May 18, compiled Aug 25 2018 20:01:39)
Included patches: 1-272
Modified by Exherbo-8.1.0272

Shougo commented 5 years ago

I have tested it but I don't reproduce the problem. I cannot work for non-reproduce able problems. Sorry.

Vim 8.1.272 seems old.

call deoplete#custom#option('num_processes', 1)

may fixes your problem.

Any suggestions on how I can debug this further once it happens again would be appreciated.

Please enable debug log.

Q: I want to debug the sources.

A: Please enable logging feature like this. >

       call deoplete#custom#option('profile', v:true)
       call deoplete#enable_logging('DEBUG', 'deoplete.log')
       call deoplete#custom#source('jedi', 'is_debug_enabled', 1)

Shougo commented 5 years ago

And I recommend for you to use neovim instead. It is better for deoplete.

hasufell commented 5 years ago

Vim 8.1.272 seems old.

Why? That is fairly new: https://www.vim.org/download.php

And I recommend for you to use neovim instead.

It is more buggy and lacks features (such as proper tooltips).

may fixes your problem.

Interesting. So there is no limit at all? Is it possible that deoplete proccesses don't get terminated properly?

I cannot work for non-reproduce able problems. Sorry.

I don't see why that is a reason to close a bug report. Not everything is easy to reproduce. That doesn't make it less of a bug.

Shougo commented 5 years ago

OK. I can re-open this. But you need to provide more information.

Why? That is fairly new: https://www.vim.org/download.php

It is old. Please check the latest version. https://github.com/vim/vim/commits/master

It is more buggy and lacks features (such as proper tooltips).

Hm. OK. But please test neovim with deoplete. If it is not occurred in neovim, it is Vim8 specific problem.

Interesting. So there is no limit at all? Is it possible that deoplete proccesses don't get terminated properly?

It may not be terminated. It does not occured in my environment though. Please provide the process information. What processes are not terminated?

It seems there are a lot of deoplete processes spawned and they accumulate memory over time. I have to kill them manually.

Please provide more information.

I don't see why that is a reason to close a bug report. Not everything is easy to reproduce. That doesn't make it less of a bug.

I don't work for non reproduce-able problems. It is my policy. deoplete is OSS. The source code is currently opened. So you can fix the problem.

And why don't you logging? I have provided the instructions.

sugarraysam commented 5 years ago

Hi, I think I may be experimenting the same issue. When I disable deoplete plugins (and sources) I do not get crashes anymore. I have tracked the problem down to deoplete this way. I have enabled logging and captured the logging output for a session that ends with a crash (see at the end). By crash I mean my computer freezes, I cannot change getty (Alt + FX), cant do anything, looks like a memory crash.

I must say that I absolutely love this plugin and it is killing me that it is crashing. You will find all my specs below. Should you need anything else just let me know. Thank you.

Here are the specs for my machine.

sugar@sugar ~> uname -a
Linux sugar 5.0.2-arch1-1-ARCH #1 SMP PREEMPT Thu Mar 14 18:47:49 UTC 2019 x86_64 GNU/Linux
sugar@sugar ~> nvim --version                                                
NVIM v0.3.4
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/bin/cc -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -Wconversion -O2 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagnostics-color=auto -Wno-array-bounds -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/build/neovim/src/build/config -I/build/neovim/src/neovim-0.3.4/src -I/usr/include -I/build/neovim/src/build/src/nvim/auto -I/build/neovim/src/build/include
Compiled by builduser

Features: +acl +iconv +jemalloc +tui 
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info
sugar@sugar ~> python --version                                                           
Python 3.7.2

Here are my neovim plugin configs:

"""
""" Define plugins (https://github.com/junegunn/vim-plug)
"""
call plug#begin('~/.local/share/nvim/plugged')

" asynchronous syntax checking (similar https://github.com/vim-syntastic/syntastic)
Plug 'vim-syntastic/syntastic' " better than neomake bcuz seamless integration /w airline

" Deoplete completion engine (was buggy as of Feb 25 2019)
Plug 'Shougo/deoplete.nvim', { 'do': ':UpdateRemotePlugins' }
 completion sources for deoplete (see https://github.com/Shougo/deoplete.nvim/wiki/Completion-Sources)
Plug 'Shougo/neco-syntax'
Plug 'zchee/deoplete-jedi'

" Others
Plug 'godlygeek/tabular'              "Align code nicely
Plug 'ervandew/supertab'              "Autocomplete with tab
Plug 'scrooloose/nerdtree'            "file management Plug
Plug 'scrooloose/nerdcommenter'       "easy commenting
Plug 'vim-airline/vim-airline'        "status line
Plug 'kien/ctrlp.vim'                 "Fuzzy file, buffer, mru, tag, etc finder.

" Initialize plugin environment
call plug#end()

Here are my deoplete settings:

"""
""" deoplete
"""
" Always use deoplete
let g:deoplete#enable_at_startup = 1
call deoplete#custom#option('auto_complete_delay', 2)
call deoplete#custom#option('smart_case', v:true)
call deoplete#custom#option('num_processes', v:true) "memory consumption problems?

" Close preview window after completion is done
autocmd CompleteDone * silent! pclose!

" Enable Debugging
call deoplete#enable_logging('DEBUG', '/home/sugar/.config/nvim/debug/deoplete.log')

" Options for deoplete/jedi (https://github.com/zchee/deoplete-jedi)
let g:deoplete#sources#jedi#server_timeout = 100
let g:deoplete#sources#jedi#show_docstring = 1
let g:deoplete#sources#jedi#python_path = '/usr/bin/python3'
let g:deoplete#sources#jedi#debug_server = 1  "for debugging

Here is the logging from the crash (I left beginning of trace and end, had to cut in the middle to fit pastebin max length constraint): View crash log on pastebin

Shougo commented 5 years ago

Here is the logging from the crash (I left beginning of trace and end, had to cut in the middle to fit pastebin max length constraint): View crash log on pastebin

I have read the log, but I don't see the reason. core dump is needed. Please upload it.

Shougo commented 5 years ago

By crash I mean my computer freezes, I cannot change getty (Alt + FX), cant do anything, looks like a memory crash.

It is really deoplete problem? Please test deoplete in other environments. I don't reproduce the problem.

sugarraysam commented 5 years ago

Hi, unfortunately the coredumps are "Missing" as you can see...

Sat 2019-03-16 08:20:41 EDT    5791  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/execve
Sat 2019-03-16 08:21:49 EDT    6302  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/execve
Sat 2019-03-16 08:21:54 EDT    6430  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/execve
Sat 2019-03-16 08:36:25 EDT    9390  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/test_shellcode
Sat 2019-03-16 11:00:56 EDT    4864  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/test
Sat 2019-03-16 11:01:12 EDT    5172  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/test
Sat 2019-03-16 16:28:38 EDT   12450  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/test
Sat 2019-03-16 16:28:42 EDT   12519  1000  1000  11 missing   /home/sugar/opt/ringzer0/shellcode/level1/test

Getting more info from a dump (they all are similar to this):

sugar@sugar ~/debug_deoplete> coredumpctl dump -o 2019_03_16_12519.coredump 12519                                                     
           PID: 12519 (test)
           UID: 1000 (sugar)
           GID: 1000 (sugar)
        Signal: 11 (SEGV)
     Timestamp: Sat 2019-03-16 16:28:42 EDT (3 days ago)
  Command Line: ./test
    Executable: /home/sugar/opt/ringzer0/shellcode/level1/test
 Control Group: /user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service                                           
          Unit: user@1000.service
     User Unit: gnome-terminal-server.service
         Slice: user-1000.slice
     Owner UID: 1000 (sugar)
       Boot ID: ****
    Machine ID: ***
      Hostname: sugar
       Storage: /var/lib/systemd/coredump/core.test.1000.9e7410a54f4943b2a73c1660731e7598.12519.1552768122000000.lz4 (inaccessible)   
       Message: Process 12519 (test) of user 1000 dumped core.

                Stack trace of thread 12519:
                #0  0x00007f65d8ab8040 n/a (n/a)
File "/var/lib/systemd/coredump

I use gnome-terminal, and every terminal is attached to the gnome-terminal-server process which is multithreaded. I've read quickly and it seems some threads could hold mutexes for long period (maybe forever). This is not really my cup of tea....what terminal are you using? Im thinking I could switch terminal and it would fix my issue.

Thank you.

Shougo commented 5 years ago

I use both alacritty and kitty.

sugarraysam commented 5 years ago

Thanks, I've noticed other coredumps linked to gnome-terminal-server, I will give a try to alacritty see if it fixes my issues. :-)

StephenPCG commented 5 years ago

I have encountered the same issue. My env:

OS: macOS 10.14.4
vim: NVIM v0.3.4
deoplete: latest (DeinUpdate just two hours ago)
deoplete-jedi: latest

After working with python code for about two hours, there are about 584 deoplete processes:

$ ps aux | grep deoplete | wc -l
584

process name reported by ps is:

$HOME/.venvs/django/bin/python $HOME/.vim/cache/dein/.cache/init.vim/.dein/rplugin/python3/deoplete/vendored/jedi/jedi/evaluate/compiled/subprocess/__main__.py $HOME/.vim/cache/dein/.cache/init.vim/.dein/rplugin/python3/deoplete/vendored/parso 3.7.3

This happens constantly with my current config, after working with python source files for some hours, there are hundreds of deoplete processes and eat up tens of GB of memory.

I will try to create a minimal config to reproduce this problem a few days later when I get spare time.

Shougo commented 5 years ago

@StephenPCG All reported users use jedi with deoplete. And the processes are jedi. So it is not deoplete problem. It is jedi problem instead.

Shougo commented 5 years ago

You can disable deoplete-jedi.

StephenPCG commented 5 years ago

@Shougo Could it be problem of jedi or deoplete-jedi?

Shougo commented 5 years ago

jedi instead of deoplete-jedi

Shougo / deoplete.nvim

Memory consumption issues #941

Problems summary

Environment Information