citre cause error "Creating pipe: Too many open files" with args (prefix)

haoisli9 commented 2 years ago

win10, emacs 29.0.50 when using company to auto-completion, it always cause error "Creating pipe: Too many open files" with args (prefix); when I exit emacs, I can see readtags processing in background, like: readtags stderr -- open readtags-stderr -- Main (serial port ?) readtags stderr<1> -- open readtags-stderr-254200 -- Main (serial port ?) readtags stderr<2> -- open readtags-stderr-376378 -- Main (serial port ?)

This problem happens recently after last citre update. Old citre version does not have this problem.

Can you check it? Thanks.

AmaiKinono commented 2 years ago

How did you get Emacs 29.0.50 on Windows? If you downloaded it somewhere, please offer me the URL; If you build it from source, please give instructions.

haoisli9 commented 2 years ago

Hi,

 I get emacs source code from https://github.com/emacs-mirror/emacs and build it by emacs-build(https://github.com/kiennq/emacs-build);  All the things can be done automaticly.
 I build it with the flag: with-all and pack-all; all the externsions are not required.

  BTW, are you Chinese? maybe we can discuss with "中国话"?

@.***

From: AmaiKinono Date: 2022-06-24 16:12 To: universal-ctags/citre CC: haoisli9; Author Subject: Re: [universal-ctags/citre] citre cause error "Creating pipe: Too many open files" with args (prefix) (Issue #129) How did you get Emacs 29.0.50 on Windows? If you downloaded it somewhere, please offer me the URL; If you build it from source, please give instructions. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

haoisli9 commented 2 years ago

hi, how about this issue? need more information?

AmaiKinono commented 2 years ago

I haven't got time to work on it.

AmaiKinono commented 2 years ago

Sadly I can't reproduce this.

I'm on

Windows 10.
Emacs 29.0.50 (downloaded from https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-29/, emacs-29.0.50-snapshot-2022-04-02.zip).
Citre is on the latest version on master.

Company configuration are just 2 lines:

(setq company-minimal-prefix-length 1)
(setq company-backends '(company-capf))

Tested in the Linux kernel. Typing is smooth, the error you mentioned didn't happen, and I didn't see any Citre related processes by list-processes or quitting Emacs.

I could suggest a patch that you can try. In citre-common.el, there's this function:

(defun citre-kill-process-buffer (buffer)
  "Delete the process in BUFFER and kill BUFFER.
This doesn't run `kill-buffer-hook' and
`kill-buffer-query-functions' so it should be faster."
  (let ((kill-buffer-hook nil)
        (kill-buffer-query-functions nil))
    (when-let ((proc (get-buffer-process buffer)))
      (delete-process proc))
    (kill-buffer buffer)))

Try change it to:

(defun citre-kill-process-buffer (buffer)
  "Delete the process in BUFFER and kill BUFFER.
This doesn't run `kill-buffer-hook' and
`kill-buffer-query-functions' so it should be faster."
  (let ((kill-buffer-hook nil)
        (kill-buffer-query-functions nil))
    (when-let ((proc (get-buffer-process buffer)))
      (signal-process proc 'sighup)
      (delete-process proc))
    (kill-buffer buffer)))

maybe we can discuss with "中国话"?

Yes, but for other maintainers' sake, I have to attach an English version of everything we are saying. I'm OK with that but using English saves developer's time.

masatake commented 2 years ago

How does M-x list-processes say when getting the error?

haoisli9 commented 2 years ago

like this：

company-tabnine--process 13332 run -- -- Main d:/Users/home/.TabNine/4.4.56/x86_64-pc-windows-gnu/TabNine.exe --client emacs readtags stderr -- open readtags-stderr -- Main (serial port ?) readtags stderr<1> -- open readtags-stderr-331043 -- Main (serial port ?) readtags stderr<2> -- open readtags-stderr-488999 -- Main (serial port ?) server -- listen -- -- Main (network server on local:12916)

@.***

From: Masatake YAMATO Date: 2022-07-03 14:54 To: universal-ctags/citre CC: haoisli9; Author Subject: Re: [universal-ctags/citre] citre cause error "Creating pipe: Too many open files" with args (prefix) (Issue #129) How does M-x list-processes say when getting the error? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

haoisli9 commented 2 years ago

It's a bad news.

I try to patch citre-common.el, and an error rasise and saying:

error in process sentinel: citre-kill-process-buffer: Cannot signal process readtags stderr<2> error in process sentinel: Cannot signal process readtags stderr<2>

The source I'am trying is ctags-master, in https://github.com/universal-ctags/ctags, when I try to type something in main.c this error happened.

maybe you can try with ctags-master source ?

thanks.

@.***

From: AmaiKinono Date: 2022-07-03 13:00 To: universal-ctags/citre CC: haoisli9; Author Subject: Re: [universal-ctags/citre] citre cause error "Creating pipe: Too many open files" with args (prefix) (Issue #129) Sadly I can't reproduce this. I'm on Windows 10. Emacs 29.0.50 (downloaded from https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-29/, emacs-29.0.50-snapshot-2022-04-02.zip). Citre is on the latest version on master. Company configuration are just 2 lines: (setq company-minimal-prefix-length 1) (setq company-backends '(company-capf)) Tested in the Linux kernel. Typing is smooth, the error you mentioned didn't happen, and I didn't see any Citre related processes by list-processes or quitting Emacs. I could suggest a patch that you can try. In citre-common.el, there's this function: (defun citre-kill-process-buffer (buffer) "Delete the process in BUFFER and kill BUFFER. This doesn't run kill-buffer-hook' and kill-buffer-query-functions' so it should be faster." (let ((kill-buffer-hook nil) (kill-buffer-query-functions nil)) (when-let ((proc (get-buffer-process buffer))) (delete-process proc)) (kill-buffer buffer))) Try change it to: (defun citre-kill-process-buffer (buffer) "Delete the process in BUFFER and kill BUFFER. This doesn't run kill-buffer-hook' and kill-buffer-query-functions' so it should be faster." (let ((kill-buffer-hook nil) (kill-buffer-query-functions nil)) (when-let ((proc (get-buffer-process buffer))) (signal-process proc 'sighup) (delete-process proc)) (kill-buffer buffer))) maybe we can discuss with "中国话"? Yes, but for other maintainers' sake, I have to attach an English version of everything we are saying. I'm OK with that but using English saves developer's time. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

masatake commented 2 years ago

readtags doesn't open much files. So I guessed too many readtags processes run at the same time. However, list-processes printed only three readtags processes.

So I guess "Creating pipe: Too many open files" implies there are too many running applications (how do you call them on Windows?) other than readtags or there are some? applications (other than readtags) opening too many files.

@AmaiKinono

I wonder why

readtags stderr --      open     *readtags-stderr*        --           Main         (serial port ?)

is kept being running. Is it doing "large" searching?

masatake commented 2 years ago

I don't know and use Windows. Is there way to list files opened now? On GNU/Linux, lsof is the tool for the purpose.

How about windows? What I found with web-searching:

https://www.nirsoft.net/utils/opened_files_view.html

haoisli9 commented 2 years ago

I never run readtags outside citre. and each time error occur, it always three readtags process. the tags file is locate in ~/.cache/tags, and the tags file is about 8M.

readtags seems not exit when searching end. I use ctags version is $ ctags --version Universal Ctags 5.9.0(787811b), Copyright (C) 2015-2022 Universal Ctags Team Universal Ctags is derived from Exuberant Ctags. Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert Compiled: Jun 29 2022, 03:25:46 URL: https://ctags.io/ Optional compiled features: +win32, +wildcards, +regex, +gnulib_regex, +internal-sort, +unix-path-separator, +iconv, +option-directory, +xpath, +json, +interactive, +yaml, +case-insensitive-filenames, +packcc, +optscript, +pcre2

@.***

From: Masatake YAMATO Date: 2022-07-04 13:55 To: universal-ctags/citre CC: haoisli9; Author Subject: Re: [universal-ctags/citre] citre cause error "Creating pipe: Too many open files" with args (prefix) (Issue #129) readtags doesn't open much files. So I guessed too many readtags processes run at the same time. However, list-processes printed only three readtags processes. So I guess "Creating pipe: Too many open files" implies there are too many running applications (how do you call them on Windows?) other than readtags or there are some? applications (other than readtags) opening too many files. @AmaiKinono I wonder why readtags stderr -- open readtags-stderr -- Main (serial port ?)

is kept being running. Is it doing "large" searching? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

AmaiKinono commented 2 years ago

@masatake Let me explain the big picture.

@haoisli9 uses company, basically you can think of it as a minor mode that calls completion-at-point-functions after every keystroke.

The capf function offered by Citre can be expensive to run (when the tags file is large). To eliminate this problem, citre-capf--get-collection wraps some time-consuming part using while-no-input (this depends on citre-capf-optimize-for-popup), so the user could always interrupt readtags process by typing, which translates to a local quit by while-no-input.

So we must do cleanup when a local quit arrives. This is done in an unwind-protect form in citre-get-output-lines, which calls citre-destruct-process and it does all the jobs: kill the process, the stderr pipe process, and the stderr buffer. It seems part of (or the whole) cleanup mechanism doesn't work for @haoisli9.

I can't reproduce this so I can't inspect it further. I have several guesses that may work or not.

Some commit on Emacs master breaks / changes the behavior of unwind-protect.
citre-destruct-process is not called. It's easy to inspect, just insert some message in it and typing with company-mode on.
The stderr pipe process is not killed. Figure out how to kill a pipe process in Windows and replace the relevant part in citre-destruct-process with it. I think you can get the process object or buffer somehow from list-processes buffer.
Maybe the company-tabnine backend is interfering with the capf backend. Disable it and try again.
When nothing works, try reproduce using $ emacs -Q. Maybe the problem will disappear and then it would be easy to figure out what's wrong.

masatake commented 2 years ago

@AmaiKinono It looks that the failures of killing processes keep too many processes alive.

error in process sentinel: citre-kill-process-buffer: Cannot signal process readtags stderr<2>

I wonder we cannot send a signal.

This message comes from https://github.com/emacs-mirror/emacs/blob/master/src/process.c#L7091 .

      pid = XPROCESS (process)->pid;
      if (pid <= 0)
    error ("Cannot signal process %s", SDATA (XPROCESS (process)->name));

https://github.com/emacs-mirror/emacs/blob/c4e93b67c456b2a7cfc57b13c5d8070eb2b6d167/src/process.h#L122

    /* Process ID.  A positive value is a child process ID.
       Zero is for pseudo-processes such as network or serial connections,
       or for processes that have not been fully created yet.
       -1 is for a process that was not created successfully.
       -2 is for a pty with no process, e.g., for GDB.  */
    pid_t pid;

AmaiKinono commented 2 years ago

I wonder why we cannot send a signal.

My assumption is we cannot signal a pipe process. They should belong to "pseudo-processes such as network or serial connections".

(process-id (make-pipe-process :name "pipe"))
;; => nil (is nil defined as 0 in C?)

Citre uses delete-process to kill the pipe process. I'm suspecting it doesn't work well on Windows (though it works well on my Windows machine). SIGHUP seems to work well on Windows on readtags processes while other methods sometimes fail, so I suggested to try SIGHUP, but seems we can't signal a pipe process.

haoisli9 commented 2 years ago

I am not a administrator on my windows system. maybe it is a permissions problem? I can not confirm it because i can not get administrator's permission.

@.***

From: AmaiKinono Date: 2022-07-04 20:14 To: universal-ctags/citre CC: haoisli9; Mention Subject: Re: [universal-ctags/citre] citre cause error "Creating pipe: Too many open files" with args (prefix) (Issue #129) I wonder why we cannot send a signal. My assumption is we cannot signal a pipe process. They should belong to "pseudo-processes such as network or serial connections". (process-id (make-pipe-process :name "pipe")) ;; => nil (is nil defined as 0 in C?)

Citre uses delete-process to kill the pipe process. I'm suspecting it doesn't work well on Windows (though it works well on my Windows machine). SIGHUP seems to work well on Windows on readtags processes while other methods sometimes fail, so I suggested to try SIGHUP, but seems we can't signal a pipe process. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

AmaiKinono commented 2 years ago

Guessing doesn't help if we can't reproduce the problem.

You are on your own. Try advices in https://github.com/universal-ctags/citre/issues/129#issuecomment-1173528189, or give a recipe that starts from $ emacs -Q and reproduces the problem.

haoisli9 commented 2 years ago

Maybe the company-tabnine backend is interfering with the capf backend. Disable it and try again. Tried setting (setq company-backends '(company-capf)) , same thing.

      using $ emacs -Q, same thing.

I tried to debug citre-common, find that when this error happened, multiple readtags process list in list-process buffer. process name is <readtags stderr<1>> and so on. and than i debug citre-get-output-lines, can not enter unwind-protect block. so the citre-destruct-process does not called.

    Some guess: 
     1、multiple readtags process is not correct? maybe windows delete-process is slow and not clean? we can check "readtags process" when call readtags? 
     2、seems unwind-protect is not handled as what we want? I use emacs 29.0.50 up to date, and same things.

@.***

From: AmaiKinono Date: 2022-07-06 19:02 To: universal-ctags/citre CC: haoisli9; Mention Subject: Re: [universal-ctags/citre] citre cause error "Creating pipe: Too many open files" with args (prefix) (Issue #129) Guessing doesn't help if we can't reproduce the problem. You are on your own. Try advices in #129 (comment), or give a recipe that starts from $ emacs -Q and reproduces the problem. ― Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

AmaiKinono commented 3 months ago

Close as developers can't reproduce the problem, and we haven't receive a reproduce guide from $ emacs -Q.

Feel free to reopen if you have a solution.

universal-ctags / citre

citre cause error "Creating pipe: Too many open files" with args (prefix) #129