bash>=4.4: bad substitution error on non-word characters

[iakremnev]

Bug Report

When I try to tab-complete dvc add [targets...] command, it can fail in the following scenario:

1. multiple targets are specified
2. the first target contains path separator

Example:

$ dvc add data/r[TAB]  # --> dvc add data/raw  # Completes OK
$ dvc add data/raw da[TAB]  # --> dvc add data/raw datbash: _shtab_dvc_add_data/raw_COMPGEN: bad substitution

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.7.9 (pip)
---------------------------------
Platform: Python 3.6.9 on Linux-5.4.0-47-generic-x86_64-with-Ubuntu-18.04-bionic
Supports: http, https, s3
Cache types: hardlink, symlink
Cache directory: ext3 on /dev/sda1
Workspace directory: ext3 on /dev/sda1
Repo: dvc, git

[efiop]

Thanks for the report @iakremnev ! Transfering to shtab.

[casperdcl]

can't reproduce this. can you post the output of shtab --version?

[iakremnev]

shtab 1.3.1

[iakremnev]

Here's how I reproduce this bug:

mkdir test && cd test
dvc init --no-scm
mkdir a
touch a/b.txt a/c.txt
dvc add a/b.txt a[TAB]
dvc add a/b.txt abash: _shtab_dvc_add_a/b.txt_COMPGEN: bad substitution

[casperdcl]

Hmm I've followed exactly your code and get:

dvc add a/b.txt a[TAB]
a   a/

Could you try re-initialising completion?

Try doing this:

eval "$(dvc completion -s bash)"
dvc add a/b.txt a[TAB]

If that works then you probably just need to re-install completions (https://dvc.org/doc/install/completion)

[iakremnev]

Re-installed with apt install --reinstall, still no luck.

Here's some stack trace with set -x:

$ dvc add data/raw d[TAB]
+ local word=d
+ COMPREPLY=()
+ '[' 3 -eq 1 ']'
+ '[' 3 -eq 2 ']'
+ '[' 3 -ge 3 ']'
+ _shtab_dvc_compgen_subcommand_ add data/raw
++ _shtab_replace_hyphen add
++ echo add
++ sed s/-/_/g
++ _shtab_replace_hyphen data/raw
++ echo data/raw
++ sed s/-/_/g
+ local flags_list=_shtab_dvc_add_data/raw
+ local args_gen=_shtab_dvc_add_data/raw_COMPGEN
bash: _shtab_dvc_add_data/raw_COMPGEN: bad substitution

[iakremnev]

The executed code from /etc/bash_completion.d/dvc is:

# Notes:
# `COMPREPLY` contains what will be rendered after completion is triggered
# `word` refers to the current typed word
# `${!var}` is to evaluate the content of `var`
# and expand its content as a variable
#       hello="world"
#       x="hello"
#       ${!x} ->  ${hello} ->  "world"
_shtab_dvc() {
  local word="${COMP_WORDS[COMP_CWORD]}"

  COMPREPLY=()

  if [ "${COMP_CWORD}" -eq 1 ]; then
    _shtab_dvc_compgen_root_ ${COMP_WORDS[1]}
  elif [ "${COMP_CWORD}" -eq 2 ]; then
    _shtab_dvc_compgen_command_ ${COMP_WORDS[1]}
  elif [ "${COMP_CWORD}" -ge 3 ]; then
    _shtab_dvc_compgen_subcommand_ ${COMP_WORDS[1]} ${COMP_WORDS[2]}
  fi

  return 0
}

It seems like ${COMP_CWORD} should be equal to 2, not 3, and _shtab_dvc_compgen_command_ should be called instead of _shtab_dvc_compgen_subcommand_. My reasoning is that dvc add is a command and not a subcommand.

[iakremnev]

According to bash man page, COMP_CWORD is exactly 3 at position dvc add data/raw da[TAB]. There's clearly a bug in these branching statements.

[casperdcl]

COMP_CWORD isn't the issue here. This error doesn't make sense:

+ local args_gen=_shtab_dvc_add_data/raw_COMPGEN
bash: _shtab_dvc_add_data/raw_COMPGEN: bad substitution

can you run this?

foo(){
  local args_gen=_shtab_dvc_add_data/raw_COMPGEN
}
foo

there's no "substitution" occurring in this command so it should just work without errors. What bash --version are you using?

[iakremnev]

Actually, it's not this line causing the error but the next one:

_shtab_dvc_compgen_subcommand_() {
  local flags_list="_shtab_dvc_$(_shtab_replace_hyphen $1)_$(_shtab_replace_hyphen $2)"
  local args_gen="${flags_list}_COMPGEN"   # <-- this is _shtab_dvc_add_data/raw_COMPGEN now
  [ -n "${!args_gen}" ] && local opts_more="$(${!args_gen} "$word")"  # fails on ${!args_gen}
  local opts="${!flags_list}"
  if [ -z "$opts$opts_more" ]; then
    _shtab_dvc_compgen_command_ $1
  else
    COMPREPLY=( $(compgen -W "$opts" -- "$word"; [ -n "$opts_more" ] && echo "$opts_more") )
  fi
}

Even smaller example:

$ foo() {
>  local x=a/b
>  [ ${!x} ]
>}
$ foo
bash: a/b: bad substitution

[iakremnev]

Let me explain once again: when I type dvc add data/raw da and press TAB, COMP_WORDS=(dvc add data/raw da) and COMP_CWORD=3. Therefore

elif [ "${COMP_CWORD}" -ge 3 ]; then
    _shtab_dvc_compgen_subcommand_ ${COMP_WORDS[1]} ${COMP_WORDS[2]}

is called as if dvc add data/raw is a valid subcommand. Are you sure it's the intended behavior?

[casperdcl]

$ foo(){
>  local x=a/b
>  [ ${!x} ]  # command exits non-zero but shouldn't cause a crash
>  echo "all fine"
> }
$ foo
all fine

is called as if dvc add data/raw is a valid subcommand.

yes that's expected. [ -n "${!args_gen}" ] will exit non-zero but not crash.

[iakremnev]

Hmm, this is interesting. This function doesn't print "all fine" for me. Bash version is 4.4.20(1)-release.

BASHOPTS=checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob:extquote:force_fignore:histappend:interactive_comments:progcomp:promptvars:sourcepath
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor

[casperdcl]

Very interesting. I'm on GNU bash, version 4.3.48(1)-release with the same BASHOPTS and SHELLOPTS.

[iakremnev]

I used docker images to try this scenario in different bash versions. I found that for bash<=4.3.48 there's no failure and foo return with error code 0, and for bash>=4.4.23 there's an immediate exit with error code 1.

[casperdcl]

could you try #23?

pip install 'shtab>=1.3.2'
eval "$(dvc completion -s bash)"
dvc add a/b.txt a[TAB]

[iakremnev]

I confirm, the problem is solved! Thanks @casperdcl!