search

kislyuk / argcomplete

Python and tab completion, better together.
https://kislyuk.github.io/argcomplete/
Apache License 2.0
1.41k stars 133 forks source link

"Unable to open fd 8 for writing, quitting" on Linux #303

Open yarikoptic opened 4 years ago

yarikoptic commented 4 years ago

Relevant older issue: https://github.com/kislyuk/argcomplete/issues/142 where I believe users also reported it happening on Linux but it was closed without any resolution (e.g. via suggested use of temporary files).

In my case I have a stock Debian (testing/unstable mix), python 3.8.3rc1 and I observe the same error with our DataLad which recently was fixed up to work (again) with argcomplete and it works for everyone else on the team but (un)lucky me. In the session you below can see that I am getting the exception, and the process has no 8 fd among its fds:

$> _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
[DEBUG  ] Command line args 1st pass. Parsed: Namespace() Unparsed: ['--pdb', '--help'] 
[DEBUG  ] Discovering plugins 
[DEBUG  ] Loading entrypoints 
[DEBUG  ] Generating detailed description for the parser 
[Level 5] Finished setup_parser 
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(188)__call__()
-> try:
(Pdb) l
183             except:
184                 debug_stream = sys.stderr
185     
186             if output_stream is None:
187                 import pdb; pdb.set_trace()
188  ->             try:
189                     output_stream = os.fdopen(8, "wb")
190                 except:
191                     debug("Unable to open fd 8 for writing, quitting")
192                     exit_method(1)
193     
(Pdb) os.getpid()
1092474
(Pdb) 
[3]  + 1092474 suspended  _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help

$> ls -l /proc/1092474/fd 
total 0
lrwx------ 1 yoh yoh 64 May 19 13:48 0 -> /dev/pts/16
lrwx------ 1 yoh yoh 64 May 19 13:48 1 -> /dev/pts/16
lrwx------ 1 yoh yoh 64 May 19 13:48 2 -> /dev/pts/16
lr-x------ 1 yoh yoh 64 May 19 13:48 3 -> pipe:[41232747]
lr-x------ 1 yoh yoh 64 May 19 13:48 5 -> pipe:[41232748]

$> fg
[3]    1092474 continued  _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --pdb -l 1 --help
l
l
194             # print("", stream=debug_stream)
195             # for v in "COMP_CWORD COMP_LINE COMP_POINT COMP_TYPE COMP_KEY _ARGCOMPLETE_COMP_WORDBREAKS COMP_WORDS".split():
196             #     print(v, os.environ[v], stream=debug_stream)
197     
198             ifs = os.environ.get("_ARGCOMPLETE_IFS", "\013")
199             if len(ifs) != 1:
200                 debug("Invalid value for IFS, quitting [{v}]".format(v=ifs))
201                 exit_method(1)
202     
203             comp_line = os.environ["COMP_LINE"]
204             comp_point = int(os.environ["COMP_POINT"])
(Pdb) n
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(189)__call__()
-> output_stream = os.fdopen(8, "wb")
(Pdb) 
OSError: [Errno 9] Bad file descriptor
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(189)__call__()
-> output_stream = os.fdopen(8, "wb")
(Pdb) 
> /home/yoh/proj/misc/argcomplete/argcomplete/__init__.py(190)__call__()
-> except:
(Pdb) fg
*** NameError: name 'fg' is not defined
(Pdb) 
Traceback (most recent call last):
  File "/home/yoh/proj/misc/argcomplete/argcomplete/__init__.py", line 189, in __call__
    output_stream = os.fdopen(8, "wb")
  File "/usr/lib/python3.8/os.py", line 1023, in fdopen
    return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor

in the shell I do get calls which do not explicitly create fd 8 failing:

$> python -c 'import os; os.fdopen(8)'     
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.8/os.py", line 1023, in fdopen
    return io.open(fd, *args, **kwargs)
OSError: [Errno 9] Bad file descriptor
$> python -c 'import os; os.fdopen(8)' 8>&1
$>
evanunderscore commented 4 years ago

This is expected; you cannot open a file descriptor if it doesn't exist. The shell wrappers explicitly create and read from file descriptor 8. We do this because argcomplete lives within arbitrary Python programs, so rather than trying to suppress all output to stdout, we simply redirect the stream to /dev/null and communicate on a different one.

Your actual problem likely has nothing to do with this. Can you set _ARC_DEBUG=1 and paste the output from an actual tab completion attempt? Please also provide your shell version and what you did to install and activate argcomplete.

yarikoptic commented 4 years ago

The shell wrappers explicitly create and read from file descriptor 8

Should it then be open when I source the shell wrapper? May be the problem is in it on my end.

Here is the wrapper I use - may be it is the source of trouble: ```shell $> cat tools/cmdline-completion # Universal completion script for DataLad with the core autogenerated by # python-argcomplete and only slighly umproved to work for ZSH if sourced under # ZSH if [ "${ZSH_VERSION:-}" != "" ]; then echo "I: Enabling support of bash completions for zsh" autoload -U compinit && compinit autoload -U bashcompinit && bashcompinit fi _python_argcomplete() { local IFS=' ' COMPREPLY=( $(IFS="$IFS" COMP_LINE="$COMP_LINE" COMP_POINT="$COMP_POINT" _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" _ARGCOMPLETE=1 "$1" 8>&1 9>&2 1>/dev/null 2>/dev/null) ) if [[ $? != 0 ]]; then unset COMPREPLY fi } complete -o nospace -o default -F _python_argcomplete "datalad" ```
But even after sourcing it I do not see fd 8 open ```shell lena:~/proj/datalad/datalad-maint $> source tools/cmdline-completion $> ls -l /proc/$$/fd/ total 0 0 lrwx------ 1 yoh yoh 64 May 21 09:49 0 -> /dev/pts/25 0 lrwx------ 1 yoh yoh 64 May 21 09:49 1 -> /dev/pts/25 0 lrwx------ 1 yoh yoh 64 May 21 09:49 2 -> /dev/pts/25 0 lrwx------ 1 yoh yoh 64 May 21 09:49 255 -> /dev/pts/25 ```

Should fd 8 be open at that point already or that would happen later? (upon attempt to complete?)

may be there is some shell setting which is in a way...?(btw -- tried with both bash and zsh)

Can you set _ARC_DEBUG=1 and paste the output from an actual tab completion attempt?

Note that in the initial report message I already have that env var set. And that is how I actually arrived to this specific issue -- by use of this env var... But here is plain attempt to complete which is not informative really:

(git)lena:~datalad/datalad-maint[maint]
$> source venv*/dev3/bin/activate                             

$> export _ARC_DEBUG=1

$> source tools/cmdline-completion            
I: Enabling support of bash completions for zsh

$> datalad ins<TAB>
no matches for: `file' or `corrections'

so nothing is shown.

yarikoptic commented 4 years ago
I have tried with a freshly regenerated wrapper which targets specific full path datalad entry point script where I explicitly added that marker on top -- the same silence ```shell $> source /tmp/datalad-argcomplete $> cat /tmp/datalad-argcomplete _python_argcomplete() { local IFS=$'\013' local SUPPRESS_SPACE=0 if compopt +o nospace 2> /dev/null; then SUPPRESS_SPACE=1 fi COMPREPLY=( $(IFS="$IFS" \ COMP_LINE="$COMP_LINE" \ COMP_POINT="$COMP_POINT" \ COMP_TYPE="$COMP_TYPE" \ _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \ _ARGCOMPLETE=1 \ _ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \ "$1" 8>&1 9>&2 1>/dev/null 2>/dev/null) ) if [[ $? != 0 ]]; then unset COMPREPLY elif [[ $SUPPRESS_SPACE == 1 ]] && [[ "$COMPREPLY" =~ [=/:]$ ]]; then compopt -o nospace fi } complete -o nospace -o default -F _python_argcomplete "/home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad" $> cat /home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad #!/home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/python # PYTHON_ARGCOMPLETE_OK # # Custom simplistic runner for DataLad. Assumes datalad module # being available. Generated by monkey patching monkey patched # setuptools. # from datalad.cmdline.main import main main() $> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad ins ```
kislyuk commented 4 years ago

It appears that the library/executable you are using is complex and performs a lot of initialization before calling argcomplete.autocomplete(). Some of these initialization steps may be interfering with the argcomplete protocol by using fd 8 or otherwise. The solution to your issue is to disable these initialization steps that your executable is running between the entry point and calling argcomplete, until argcomplete starts working.

evanunderscore commented 4 years ago

That looks like an old wrapper. What version of argcomplete are you running? How did you generate it?

You can try manually removing 1>/dev/null 2>/dev/null from the line where it runs your program to see if it outputs anything when you attempt to tab complete it.

Have you been able to get any other simple test programs working with argcomplete?

yarikoptic commented 4 years ago

THANK YOU @evanunderscore for the guidance -- I am ready to declare "success" in troubleshooting !

What version of argcomplete are you running?

since it is argcomplete which generates it, may be it should embed argcomplete version which generated it somewhere in the comment

TL;DR summary: it is interaction between 

_python_argcomplete() {
    local IFS=$'\013'
    local SUPPRESS_SPACE=0
    if compopt +o nospace 2> /dev/null; then
        SUPPRESS_SPACE=1
    fi
    COMPREPLY=( $(IFS="$IFS" \
                  COMP_LINE="$COMP_LINE" \
                  COMP_POINT="$COMP_POINT" \
                  COMP_TYPE="$COMP_TYPE" \
                  _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \
                  _ARGCOMPLETE=1 \
                  _ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \
                  __python_argcomplete_run "$1") )
$> IFS=$'\013' /usr/lib/git-annex.linux/git version
[1]    1039928 segmentation fault (core dumped)  IFS=$'\013' /usr/lib/git-annex.linux/git version

That also explains why other @datalad/developers didn't experience this -- since they typically do not use standalone git-annex build.

I guess there is nothing wrong done on argcomplete side, so feel free to close this issue, unless you see that you could avoid setting IFS and set some other "argcomplete specific" variable so it might not cause disturbance in some weak tools underneath?

Some additional notes which are no longer pertinent probably > How did you generate it? re-doing everything now after `pip install --upgrade argcomplete` to 1.12.0: ``` $> venvs/dev3/bin/register-python-argcomplete --shell bash `which datalad` >| /tmp/datalad-argcomplete $> bash # since I was in zsh $> which datalad $> source /tmp/datalad-argcomplete ``` and verified that it still doesn't work by trying to complete `datalad inst` > You can try manually removing 1>/dev/null 2>/dev/null from the line ... ``` *$> sed -e 's, 1>/dev/null 2>&1,,g' /tmp/datalad-argcomplete > /tmp/datalad-argcomplete-noswallow $> diff /tmp/datalad-argcomplete /tmp/datalad-argcomplete-noswallow 20c20 < "$@" 8>&1 9>&2 1>/dev/null 2>&1 --- > "$@" 8>&1 9>&2 ```
evanunderscore commented 4 years ago

Thanks for the detailed notes @yarikoptic. Glad you managed to find the source of your problem!

Embedding the argcomplete version in the generated wrapper is a good idea, though currently argcomplete doesn't have a __version__ and keeping one in sync with the package version number is a workflow problem that would need to be left to @kislyuk.