zsh completion error - invalid regex : )

[Arnatious]

Problem

Using the latest (c153f72) version of _beet still produces the invalid regex error.

$ beet …                                                                            
_beet:zregexparse:4: invalid regex : )                                                                      
_beet:zregexparse:4: invalid regex : ) 

I have mawk installed, which according to https://github.com/beetbox/beets/issues/3546#issuecomment-1878395666 should have avoided this issue in the previous version, but it's present in both versions.

$ awk -W version
mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       srandom/random
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer     2147483647

Setup

• OS: Ubuntu 22.04.3 LTS
• Python version: python 3.10.12
• beets version: 1.6.0
• Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

directory: /media/library/Music
library: /media/library/Music/musiclibrary.db
import:
  move: yes
plugins:
  chroma:
  fuzzy:
  embedart:
  fetchart:
acoustid:
  apikey: REDACTED

---

$ which _beet
_beet () {
        local _ra_p1 _ra_p2 _ra_left _ra_right _ra_com expl tmp nm="$compstate[nmatches]"
        local _ra_actions _ra_line="${(pj:\0:)${(@)words[1,CURRENT - 1]:Q}}"$'\0'"$PREFIX"
        _ra_actions=()
        zregexparse -c _ra_p1 _ra_p2 "$_ra_line" $'/[^\0]##\0/' $'(' $'(' $'/-c\0/' $'/[^\0]##\0/' $':_ra_comp $\'file:file:_files\'' $'|' $'/-v\0/' $'|' $'/-l\0/' $'/[^\0]##\0/' $':_ra_comp $\'file:file:_files\'' $'|' $'/-h\0/' $'|' $'/-d\0/' $'/[^\0]##\0/' $':_ra_comp $\'dir:directory:_dirs\'' $'|' $'/[]/' $':_ra_comp $\'options:global options:(( -c\\\\:path\\\\ to\\\\ configuration\\\\ file -v\\\\:print\\\\ debugging\\\\ information -l\\\\:library\\\\ database\\\\ file\\\\ to\\\\ use -h\\\\:show\\\\ this\\\\ help\\\\ message\\\\ and\\\\ exit -d\\\\:destination\\\\ music\\\\ directory ))\'' $')' $'#' $')' $')'
        case "$?" in
                (0 | 2) _message "no more arguments" ;;
                (1) if [[ "$_ra_line[_ra_p1 + 1, -1]" = *$'\0'* ]]
                        then
                                _message "parse failed before current word"
                        else
                                _ra_left="$_ra_line[_ra_p1 + 1, _ra_p2]"
                                _ra_right="$_ra_line[_ra_p2 + 1, -1]"
                                compset -p $(( $#PREFIX - $#_ra_line + $_ra_p1 ))
                                (( $#_ra_actions )) && _alternative "$_ra_actions[@]"
                        fi ;;
                (3) _message "invalid regex" ;;
        esac
        [[ nm -ne "$compstate[nmatches]" ]]
}

[Arnatious]

Tried on a different machine (22.04 on WSL) - found the same error happened there because I accidentally installed "beet", though it appeared after throwing some errors for missing commands - it works fine on that machine now. The first machine does not have the incorrect package.

The working which _beet looks different - a huge chunk of regexparse is missing in the broken one

Working which _beet:

$ which _beet
_beet () {
        local _ra_p1 _ra_p2 _ra_left _ra_right _ra_com expl tmp nm="$compstate[nmatches]"
        local _ra_actions _ra_line="${(pj:\0:)${(@)words[1,CURRENT - 1]:Q}}"$'\0'"$PREFIX"
        _ra_actions=()
        zregexparse -c _ra_p1 _ra_p2 "$_ra_line" $'/[^\0]##\0/' $'(' $'(' $'/-c\0/' $'/[^\0]##\0/' $':_ra_comp $\'file:file:_files\'' $'|' $'/-v\0/' $'|' $'/-l\0/' $'/[^\0]##\0/' $':_ra_comp $\'file:file:_files\'' $'|' $'/-h\0/' $'|' $'/-d\0/' $'/[^\0]##\0/' $':_ra_comp $\'dir:directory:_dirs\'' $'|' $'/[]/' $':_ra_comp $\'options:global options:(( -c\\\\:path\\\\ to\\\\ configuration\\\\ file -v\\\\:print\\\\ debugging\\\\ information -l\\\\:library\\\\ database\\\\ file\\\\ to\\\\ use -h\\\\:show\\\\ this\\\\ help\\\\ message\\\\ and\\\\ exit -d\\\\:destination\\\\ music\\\\ directory ))\'' $')' $'#' $')' $'(' $'/config\0/' $':_ra_comp $\'subcmds:subcommands:((config:show\\\\ or\\\\ edit\\\\ the\\\\ user\\\\ configuration))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd config}\'' $')' $'#' $'|' $'/fields\0/' $':_ra_comp $\'subcmds:subcommands:((fields:show\\\\ fields\\\\ available\\\\ for\\\\ queries\\\\ and\\\\ format\\\\ strings))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd fields}\'' $')' $'#' $'|' $'/help\0/' $':_ra_comp $\'subcmds:subcommands:((help:give\\\\ detailed\\\\ help\\\\ on\\\\ a\\\\ specific\\\\ sub-command))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd help}\'' $')' $'#' $'|' $'/import\0/' $':_ra_comp $\'subcmds:subcommands:((import:import\\\\ new\\\\ music))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd import}\'' $')' $'#' $'|' $'/list\0/' $':_ra_comp $\'subcmds:subcommands:((list:query\\\\ the\\\\ library))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd list}\'' $')' $'#' $'|' $'/modify\0/' $':_ra_comp $\'subcmds:subcommands:((modify:change\\\\ metadata\\\\ fields))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd modify}\'' $')' $'#' $'|' $'/move\0/' $':_ra_comp $\'subcmds:subcommands:((move:move\\\\ or\\\\ copy\\\\ items))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd move}\'' $')' $'#' $'|' $'/remove\0/' $':_ra_comp $\'subcmds:subcommands:((remove:remove\\\\ matching\\\\ items\\\\ from\\\\ the\\\\ library))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd remove}\'' $')' $'#' $'|' $'/stats\0/' $':_ra_comp $\'subcmds:subcommands:((stats:show\\\\ statistics\\\\ about\\\\ the\\\\ library\\\\ or\\\\ a\\\\ query))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd stats}\'' $')' $'#' $'|' $'/update\0/' $':_ra_comp $\'subcmds:subcommands:((update:update\\\\ the\\\\ library))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd update}\'' $')' $'#' $'|' $'/version\0/' $':_ra_comp $\'subcmds:subcommands:((version:output\\\\ version\\\\ information))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd version}\'' $')' $'#' $'|' $'/write\0/' $':_ra_comp $\'subcmds:subcommands:((write:write\\\\ tag\\\\ information\\\\ to\\\\ files))\'' $'(' $'/[^\0]##\0/' $':_ra_comp $\'option:option:{_beet_subcmd write}\'' $')' $'#' $')'
        case "$?" in
                (0 | 2) _message "no more arguments" ;;
                (1) if [[ "$_ra_line[_ra_p1 + 1, -1]" = *$'\0'* ]]
                        then
                                _message "parse failed before current word"
                        else
                                _ra_left="$_ra_line[_ra_p1 + 1, _ra_p2]"
                                _ra_right="$_ra_line[_ra_p2 + 1, -1]"
                                compset -p $(( $#PREFIX - $#_ra_line + $_ra_p1 ))
                                (( $#_ra_actions )) && _alternative "$_ra_actions[@]"
                        fi ;;
                (3) _message "invalid regex" ;;
        esac
        [[ nm -ne "$compstate[nmatches]" ]]
}

[skymoo]

Same here with beets-1.6.0-4 from Debian Bookworm, my awk is also mawk, I also updated the _beet to the version from #5081 and still the same error:

$ beet
_beet:zregexparse:4: invalid regex : )
_beet:zregexparse:4: invalid regex : )
_beet:zregexparse:4: invalid regex : )
$ beet

[mthies-unibi]

To clarify, the invalid regex : ) occurs, if no sub-commands of beet are identified when the completion code tries to parse the output of beet help. On my machine this was initially caused by another awk (not mawk) failing to match lines with at most 3 leading spaces. Now, (almost) any somewhat recent awk implementation should work.

You can check whether this works on your machine by executing this snippet from the completion code in the shell directly:

beet help | awk -v SEP=" " -v ARG2="3" -v START="Commands:" -v END2="" 'BEGIN {if(START==""){f=1}{f=0};
         if(ARG2 ~ "^[0-9]+"){LINE1 = "^[[:space:]]{0,"ARG2"}[^[:space:]]"}else{LINE1 = ARG2}}
         ($0 ~ END2 && f>0 && END2!="") {exit}
         ($0 ~ START && f<1) {f=1; if(length(START)!=0){next}}
         ($0 ~ LINE1 && f>0) {if(f<2){f=2; printf("%s",$0)}else{printf("\n%s",$0)}; next}
         (f>1) {gsub(/^[[:space:]]+|[[:space:]]+$/,"",$0); printf("%s%s",SEP, $0); next}
         END {print ""}'

This should output a non-empty list like this:

  clearart          remove images from file metadata
  config            show or edit the user configuration
  …

The exact contents of the list may be different for you, depending on the plugins you have installed.

If sub-commands are listed correctly in the shell, you should check that the completion code is loaded from the correct location. In a freshly started shell (before invoking or completing the beet command even once), which _beet should output a builtin autoload with the path to the _beet file actually used. When this file is loaded during the first invocation of the _beet shell function, the above awk-based parsing happens and the longish zregexparse command is generated from that output.

[skymoo]

That produces an empty list on my machine:

$ beet help | awk -v SEP=" " -v ARG2="3" -v START="Commands:" -v END2="" 'BEGIN {if(START==""){f=1}{f=0};
         if(ARG2 ~ "^[0-9]+"){LINE1 = "^[[:space:]]{0,"ARG2"}[^[:space:]]"}else{LINE1 = ARG2}}
         ($0 ~ END2 && f>0 && END2!="") {exit}
         ($0 ~ START && f<1) {f=1; if(length(START)!=0){next}}
         ($0 ~ LINE1 && f>0) {if(f<2){f=2; printf("%s",$0)}else{printf("\n%s",$0)}; next}
         (f>1) {gsub(/^[[:space:]]+|[[:space:]]+$/,"",$0); printf("%s%s",SEP, $0); next}
         END {print ""}'

$

and my awk is mawk:

$ awk -W version
mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       srandom/random
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer     2147483647
$ 

So now I guess I've got to figure out why it's not producing any output.

[mthies-unibi]

Here is a suggestion, how to narrow down the cause of the problem.

First, test that the output of beet help includes at the end a section like this:

...
Commands:
  ...
  unimported        list all files in the library folder which are not listed
                    in the beets library database
  update (upd, up)  update the library

The awk code ignores anything before the line Commands:, so it is vital that such a line is present. The text following this line is mostly copied verbatim, except that lines indented by more than 3 spaces are unfolded and joined to the end of the preceding line. In the example output above this would apply to the second line of help text for the unimported sub-command.

Next, you could check, if a simpler regular expression to distinguish deeply indented lines works better for you. The following shell command sets ARG2 to a very simple regexp that detects lines starting with exactly two spaces (followed by a non-space character).

beet help | awk -v SEP=" " -v ARG2="^  [^ ]" -v START="Commands:" -v END2="" 'BEGIN {if(START==""){f=1}{f=0};
         if(ARG2 ~ "^[0-9]+"){LINE1 = "^[[:space:]]{0,"ARG2"}[^[:space:]]"}else{LINE1 = ARG2}}
         ($0 ~ END2 && f>0 && END2!="") {exit}
         ($0 ~ START && f<1) {f=1; if(length(START)!=0){next}}
         ($0 ~ LINE1 && f>0) {if(f<2){f=2; printf("%s",$0)}else{printf("\n%s",$0)}; next}
         (f>1) {gsub(/^[[:space:]]+|[[:space:]]+$/,"",$0); printf("%s%s",SEP, $0); next}
         END {print ""}'

This should output the help text from above with only a single (long) line for each sub-command.

[skymoo]

That second regex produces output:

$ beet help | awk -v SEP=" " -v ARG2="^  [^ ]" -v START="Commands:" -v END2="" 'BEGIN {if(START==""){f=1}{f=0};
         if(ARG2 ~ "^[0-9]+"){LINE1 = "^[[:space:]]{0,"ARG2"}[^[:space:]]"}else{LINE1 = ARG2}}
         ($0 ~ END2 && f>0 && END2!="") {exit}
         ($0 ~ START && f<1) {f=1; if(length(START)!=0){next}}
         ($0 ~ LINE1 && f>0) {if(f<2){f=2; printf("%s",$0)}else{printf("\n%s",$0)}; next}
         (f>1) {gsub(/^[[:space:]]+|[[:space:]]+$/,"",$0); printf("%s%s",SEP, $0); next}
         END {print ""}'
  clearart          remove images from file metadata
  config            show or edit the user configuration
  embedart          embed image files into file metadata
  extractart        extract an image from file metadata
  fetchart          download album art
  fields            show fields available for queries and format strings
  ftintitle         move featured artists to the title field
  help (?)          give detailed help on a specific sub-command
  import (imp, im)  import new music
  lastgenre         fetch genres
  list (ls)         query the library
  modify (mod)      change metadata fields
  move (mv)         move or copy items
  remove (rm)       remove matching items from the library
  stats             show statistics about the library or a query
  update (upd, up)  update the library
  version           output version information
  write             write tag information to files
$ 

[mthies-unibi]

That was helpful in identifying the root cause of the problem!

According to https://github.com/ThomasDickey/mawk-snapshots/blame/master/man/TODO only mawk 1.3.4 20200717 and later support the brace operator when built with the internal regex-functions (as is usually done). This explains why your slightly older mawk (dated 20200120) does not recognize any sub-commands in the help output.

Before I submit another pull request for the zsh completion code, you could test my proposed fix by updating the installed file _beet to this patched version: https://github.com/mthies-unibi/beets/raw/mthies-unibi-patch-for-older-mawk/extra/_beet

The fix basically replaces the former [[:space:]]{0,3} with the equivalent, but more verbose [[:space:]]?[[:space:]]?[[:space:]]? to avoid the use of the brace operator.

[skymoo]

Unless I'm missing something, which is extremely likely, it doesn't seem to help?

$ wget https://github.com/mthies-unibi/beets/raw/mthies-unibi-patch-for-older-mawk/extra/_beet
--2024-03-26 10:49:51--  https://github.com/mthies-unibi/beets/raw/mthies-unibi-patch-for-older-mawk/extra/_beet
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/mthies-unibi/beets/mthies-unibi-patch-for-older-mawk/extra/_beet [following]
--2024-03-26 10:49:52--  https://raw.githubusercontent.com/mthies-unibi/beets/mthies-unibi-patch-for-older-mawk/extra/_beet
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8000::154, 2606:50c0:8001::154, 2606:50c0:8002::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8000::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10198 (10.0K) [text/plain]
Saving to: ‘_beet’

_beet                               100%[===================================================================>]   9.96K  --.-KB/s    in 0s      

2024-03-26 10:49:52 (103 MB/s) - ‘_beet’ saved [10198/10198]

$ sudo mv /usr/share/zsh/vendor-completions/_beet ~/_beet.orig
$ sudo cp _beet /usr/share/zsh/vendor-completions
$ beet import                                                                                                  
_beet:zregexparse:4: invalid regex : )
_beet:zregexparse:4: invalid regex : )
_beet:zregexparse:4: invalid regex : )
$ beet import

[mthies-unibi]

It is probably necessary to launch a fresh shell after replacing the _beet file. You could either open a new window (or tab or pane) in your terminal program or just explicitly reload the new function definition:

unfunction _beet
autoload -R -Uz _beet

Completion functions are usually auto-loaded by zsh on the first attempt to use them and changes to the definition of the functions are then not detected and picked up afterwards.

[skymoo]

Exactly the same after running the above and rebooting the machine.

[mthies-unibi]

Then let's check whether the updated regex works, if directly used in the shell:

beet help | awk -v SEP=" " -v ARG2='^[[:space:]]?[[:space:]]?[[:space:]]?[^[:space:]]' \
         -v START="Commands:" -v END2="" 'BEGIN {if(START==""){f=1}{f=0};
         if(ARG2 ~ "^[0-9]+"){LINE1 = "^[[:space:]]{0,"ARG2"}[^[:space:]]"}else{LINE1 = ARG2}}
         ($0 ~ END2 && f>0 && END2!="") {exit}
         ($0 ~ START && f<1) {f=1; if(length(START)!=0){next}}
         ($0 ~ LINE1 && f>0) {if(f<2){f=2; printf("%s",$0)}else{printf("\n%s",$0)}; next}
         (f>1) {gsub(/^[[:space:]]+|[[:space:]]+$/,"",$0); printf("%s%s",SEP, $0); next}
         END {print ""}'

This should list all sub-commands of beet with one line per sub-command.

If this works, the next step would be to check the output of echo $fpath if the function _beet might be loaded from another unexpected location. There is also echo $functions_source[_beet] to show the path of the file defining the function, but that rarely shows credible output in my testing.

[skymoo]

It works directly from the shell, as in the output is the list of subcommands with the short description.

AFAICT the only _beet is in /usr/share/zsh/vendor-completions/_beet

$ sudo find / -name _beet
/usr/share/zsh/vendor-completions/_beet
$ 

There are several paths listed, but from the above I don't think it's getting loaded from another location?

$ echo $fpath                  
/home/ram/.oh-my-zsh/plugins/git /home/ram/.oh-my-zsh/functions /home/ram/.oh-my-zsh/completions /home/ram/.oh-my-zsh/cache/completions /usr/local/share/zsh/site-functions /usr/share/zsh/vendor-functions /usr/share/zsh/vendor-completions /usr/share/zsh/functions/Calendar /usr/share/zsh/functions/Chpwd /usr/share/zsh/functions/Completion /usr/share/zsh/functions/Completion/AIX /usr/share/zsh/functions/Completion/BSD /usr/share/zsh/functions/Completion/Base /usr/share/zsh/functions/Completion/Cygwin /usr/share/zsh/functions/Completion/Darwin /usr/share/zsh/functions/Completion/Debian /usr/share/zsh/functions/Completion/Linux /usr/share/zsh/functions/Completion/Mandriva /usr/share/zsh/functions/Completion/Redhat /usr/share/zsh/functions/Completion/Solaris /usr/share/zsh/functions/Completion/Unix /usr/share/zsh/functions/Completion/X /usr/share/zsh/functions/Completion/Zsh /usr/share/zsh/functions/Completion/openSUSE /usr/share/zsh/functions/Exceptions /usr/share/zsh/functions/MIME /usr/share/zsh/functions/Math /usr/share/zsh/functions/Misc /usr/share/zsh/functions/Newuser /usr/share/zsh/functions/Prompts /usr/share/zsh/functions/TCP /usr/share/zsh/functions/VCS_Info /usr/share/zsh/functions/VCS_Info/Backends /usr/share/zsh/functions/Zftp /usr/share/zsh/functions/Zle
$

As you say, echo $functions_source[_beet] doesn't show anything helpful...

$ echo $functions_source[_beet]
/usr/share/zsh/functions/Completion/Base/_regex_arguments
$

At least I don't think it is.

[skymoo]

I tried installing mawk from Trixie, which is a lot more up to date:

$ mawk -W version
mawk 1.3.4 20240123
Copyright 2008-2023,2024, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

random-funcs:       arc4random_stir/arc4random
regex-funcs:        internal

compiled limits:
sprintf buffer      8192
maximum-integer     9223372036854775808
$ 

and later than the 20200717 verison that you mention above, yet that has the same problem with both the package provided _beet and your updated version.

[skymoo]

Also updating the whole machine to Trixie results in the same behaviour...

This machine is just a small VM that I use for managing my music library, initially I used an Arch VM but got fed up with the constant updates. As such I switched over to Debian for a more stable base, however I only ran into this issue on Debian. As such I'm wondering if I should deal with the updates on Arch as I didn't run into this issue there...

[mthies-unibi]

So the regex for the sub-commands seems to be ok and I also do not see any possibility to load another, unexpected _beet file.

When searching my home directory for relevant dot-files, I have found a cached copy of the generated zregexparse for the sub-commands in the file ${ZDOTDIR:-$HOME}/.zcompcache/beetslist. Perhaps this cached copy can get out-of-sync under some circumstances and still contains the old broken regex based on the empty list of sub-commands.

I have no specific idea, how this could happen, but since the file is regenerated on demand, it cannot hurt to delete any cached files related to completion of the beet command. In my case, there were four such files: beetslist, beetscmds, beetslibrary, beetsimport. After deleting these files and launching a fresh zsh in a new terminal window, the first attempt to complete beet im (to beet import) took a second or two and recreated the cached files.

[rastangineer]

Unless I'm missing something, which is extremely likely, it doesn't seem to help?

$ wget https://github.com/mthies-unibi/beets/raw/mthies-unibi-patch-for-older-mawk/extra/_beet
--2024-03-26 10:49:51--  https://github.com/mthies-unibi/beets/raw/mthies-unibi-patch-for-older-mawk/extra/_beet
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/mthies-unibi/beets/mthies-unibi-patch-for-older-mawk/extra/_beet [following]
--2024-03-26 10:49:52--  https://raw.githubusercontent.com/mthies-unibi/beets/mthies-unibi-patch-for-older-mawk/extra/_beet
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8000::154, 2606:50c0:8001::154, 2606:50c0:8002::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8000::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10198 (10.0K) [text/plain]
Saving to: ‘_beet’

_beet                               100%[===================================================================>]   9.96K  --.-KB/s    in 0s      

2024-03-26 10:49:52 (103 MB/s) - ‘_beet’ saved [10198/10198]

$ sudo mv /usr/share/zsh/vendor-completions/_beet ~/_beet.orig
$ sudo cp _beet /usr/share/zsh/vendor-completions
$ beet import                                                                                                  
_beet:zregexparse:4: invalid regex : )
_beet:zregexparse:4: invalid regex : )
_beet:zregexparse:4: invalid regex : )
$ beet import

I had same issue, following those commands fixed it, except that I reopen the terminal before beet import