#777 breaks the ability to add a definition for a new language or file type

[jhradilek]

After merging #777, it is no longer possible to add a definition for a new language or file type as documented in doc/tagbar.txt and Vim produces the following error:

"sample.adoc" 41L, 2665B
Tagbar: Could not execute ctags for /home/jhradilek/sample.adoc!
Executed command: "'ctags' '--extra=' '--file-scope=yes' '-f' '-' '--format=2' '--excmd=pattern
' '--fields=nksSafet' '--sort=no' '--append=no' '--language-force=asciidoc' '--asciidoc-kinds=s
iIvaA' '--options=/home/jhradilek/.vim/pack/my-plugins/start/vim-asciidoc/ctags/asciidoc.cnf' '
/tmp/vMgKbzE/4.adoc'"
Command output:
ctags: Warning: Unsupported parameter 'e' for "fields" option
ctags: Unknown language "asciidoc" in "language-force" option
Exit code: 1

The problem is that tagbar now attempts to run --language-force=asciidoc before asciidoc is defined with the --options option:

$ ctags --language-force=asciidoc --options=asciidoc.cnf sample.adoc
ctags: Unknown language "asciidoc" in "language-force" option

Moving --language-force=asciidoc after --options resolves this problem:

$ ctags --options=asciidoc.cnf --language-force=asciidoc sample.adoc

I am using Exuberant Ctags 5.8:

$ ctags --version
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jan 26 2021, 00:00:00
  Addresses: <dhiebert@users.sourceforge.net>, http://ctags.sourceforge.net
  Optional compiled features: +wildcards, +regex

Moving the contents of the asciidoc.cnf file to ~/.ctags of course solves this problem, but I would like to keep the configuration confined to my Vim plugin if at all possible.

[raven42]

Are you able to provide the asciidoc.cnf file and sample.adoc file as well so I can test this? For the fix in #777, we specifically needed to change the order of the arguments, so I'd like to come up with a solution to handle both. I'm wondering if there can be two instances of --options in the ctags command. Then we'd have to have a way to differentiate between the different option definitions in the tagbar config.

[jhradilek]

Hello, and thank you for your quick response! The asciidoc.cnf file is part of my vim-asciidoc repo, I pretty much followed the instructions from doc/tagbar.txt as much as possible:

• asciidoc.cnf
• asciidoc.vim

I for testing purposes, I added a simple README file to my repo which allows you to see the table of contents in Tagbar:

• README.adoc

[raven42]

Asciidoc case

Hmm... I'm running into an issue because my version of ctags has built-in support for the asciidoc format. I was able to repo by adding a new filetype of asciidoc2 with the definitions you gave. However, to provide a quick fix, one option for you @jhradilek would be to use the default built-in asciidoc support in ctags and then you wouldn't have a need of the custom definition.

host ~$ ctags --list-languages | grep Asciidoc
Asciidoc
host ~$ ctags --list-kinds=asciidoc
c  chapters
s  sections
S  level 2 sections
t  level 3 sections
T  level 4 sections
u  level 5 sections
a  anchors
host ~$ ctags --version
Universal Ctags 5.9.0(528f61f2), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jan 19 2021, 11:33:09
  URL: https://ctags.io/
  Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +packcc
host ~$

Long term issue

I can see arguments for both cases here. After some testing, I don't see an easy way to handle both cases cleanly. We would almost have to parse the contents of the deffile to check for different parameters to determine if they should go before or after the --language-force option. Or alternately we could add a new tagbar config in the type definition to include a different file post the language definition.

Case 1 - override --kinds

On the one hand with #776, we want to override the default --kinds-<lang> definition, to change the behavior. In this instance the deffile needs to be done after setting the language definition. If the deffile gets included before the language definition, then the default language definition gets set and the deffile becomes useless. Now to handle this case, there could be an alternative solution to override the default perl definition in the .vimrc like this:

    let g:tagbar_type_perl = {
                 \ 'kinds' : [
                     \ 'c:constants:0:0',
                     \ 'f:formats:0:0',
                     \ 'l:labels:0:1',
                  "  \ 'p:packages:1:0',
                     \ 's:subroutines:0:1',
                     \ 'd:subroutineDeclaration:0:0',
                     \ 'M:modules:0:0',
                 \ ],
             \ }

This will provide the same basic perl definition, but override the kinds to remove the -p option as in the case of #776. So this could be a solution to revert the #777 change but still allow that issue to be fixed (albeit a little more code needed).

Case 2 - custom language definition

In this case, defining a custom language definition now seems broken if you follow the documentation. In this instance the deffile needs to come before the --language-force option so ctags knows the language when trying to set the language kinds and definition. Now again here there is an alternative solution of putting the <lang>.cnf file in the ~/.ctags.d/ path and not include the deffile requirement from tagbar. This would cause ctags to load the file definition before any command line arguments are parsed, so the --language-force will work fine here (this is actually how I have my environment setup for a couple custom language parsers).

Alternate Case 3 - hybrid approach

Now in my testing, I was able to get both to work by using two separate --option directives. Basically by doing something like this:

ctags --extras=+F -f - --format=2 --excmd=pattern --fields=nksSafet --sort=no --append=no -V --options=ctags/asciidoc.cnf --language-force=asciidoc2 --asciidoc2-kinds=siIvaA --options=ctags/asciidoc-post.cnf README.newadoc

Then this could work. There would be one file asciidoc.cnf which is include prior to the --language-force option, and a second asciidoc-post.cnf that would be included after the --language-force option. To implement this would require a new config in the tagbar definition. Maybe something like a deffile_post which could be used for the Case 1 instances?

Summary

It is hard to say which one is the right approach. Case 2 at least is documented to use the deffile, so there might be a stronger case for reverting #777, however that will make custom override cases like in Case 1 more cumbersome. In both cases there are alternate solutions and unfortunately it is hard to say how many people might be using each approach.

This is a difficult one to go with because there are many different use-cases and I'm not sure we can account for them all. We try to be flexible to support as many options as people would want to use, but in some cases we have conflicting use-cases like this and adding options to support both can lead to complexities and make the configuration harder to maintain. So just adding additional options isn't always the best either as it makes the plugins harder to use because you have more try to figure out to get it to work how you want.

@alerque as one of the other maintainers, do you have any thoughts as to the best approach for this?

[jhradilek]

Thank you for a very detailed analysis and explanation and yeah, I see how this is a tough choice. Unfortunately, my version of ctags (version 5.8 on both latest Fedora 34 and Red Hat Enterprise Linux 8.4) does not have built-in support for Asciidoc:

$ ctags --list-languages | grep -i ascii
$ ctags --list-kinds=asciidoc
ctags: Unknown language "asciidoc" in "list-kinds" option

Obviously I would love to be able to add support to any language from within Vim configuration but for now, I will just move the contents of asciidoc.cnf to ~/.ctags.

[AndyMcAliley]

It looks like this issue remains. I encountered the problem recently in an Ultisnips .snippets file. I'm wondering if it would work to do a simplified version of case 3: use the --option directive twice, but provide the same deffile both times. It solves my specific use case, but I don't know if that solution would work more generally. Just a thought!