Making reference tags for external entities instead of definitions tags

masatake commented 4 years ago

Some parsers capturing the tokens referring to external entities as definitions tags. This should not[1]. Instead, it should capture as reference tags. The modifications for solving this issue may break compatibility. However, implementing multi-pass parsing in the future, we must fix them now.

let's call "tags referring to external entities" "dependency references" or "depedeny reference tags" as proposed in https://github.com/universal-ctags/ctags/pull/3535#issuecomment-1320710076.

[ ] using in C++
[x] #include in C/C++
[x] import in Python
[x] \input and \include in Tex
[x] require/require_relative/load in Ruby
[ ] autoload in Ruby
[x] use in Perl
[ ] import in JavsScript
[x] source and . in Shell
[ ] `include in SystemVerilog
[x] import in Protobuf
[x] import in Go
[x] include:: in ReStructuredText
[ ] include::in asciidoc ...

[1] The following question illustrates the reason: https://stackoverflow.com/questions/3609433/skip-python-import-statements-in-exuberant-ctags

masatake commented 4 years ago

If external entities are captured with proper kinds and proper roles, we can implement semantic recursion feature.

I have implemented such feature as a prototype (-S option):

[yamato@slave ctags-github]$ u-ctags --kinds-python=F --sort=no -o - --extras=+r -S /usr/lib/python3.6/site-packages/pygments/cmdline.py  
u-ctags --kinds-python=F --sort=no -o - --extras=+r -S /usr/lib/python3.6/site-packages/pygments/cmdline.py  
__main__    /usr/lib/python3.6/site-packages/pygments/__init__.py   /^if __name__ == '__main__':  # pragma: no cover$/;"    m   language:PythonMain
codetagify  /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'codetagify':     CodeTagFilter,$/;"  f   language:ansibleFilter  pyexpr:CodeTagFilter
keywordcase /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'keywordcase':    KeywordCaseFilter,$/;"  f   language:ansibleFilter  pyexpr:KeywordCaseFilter
highlight   /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'highlight':      NameHighlightFilter,$/;"    f   language:ansibleFilter  pyexpr:NameHighlightFilter
raiseonerror    /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'raiseonerror':   RaiseOnErrorTokenFilter,$/;"    f   language:ansibleFilter  pyexpr:RaiseOnErrorTokenFilter
whitespace  /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'whitespace':     VisibleWhitespaceFilter,$/;"    f   language:ansibleFilter  pyexpr:VisibleWhitespaceFilter
gobble  /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'gobble':         GobbleFilter,$/;"   f   language:ansibleFilter  pyexpr:GobbleFilter
tokenmerge  /usr/lib/python3.6/site-packages/pygments/filters/__init__.py   /^    'tokenmerge':     TokenMergeFilter,$/;"   f   language:ansibleFilter  pyexpr:TokenMergeFilter
__main__    /usr/lib/python3.6/site-packages/pygments/lexers/_mapping.py    /^if __name__ == '__main__':  # pragma: no cover$/;"    m   language:PythonMain
__main__    /usr/lib/python3.6/site-packages/pygments/formatters/_mapping.py    /^if __name__ == '__main__':  # pragma: no cover$/;"    m   language:PythonMain

As input file, only /usr/lib/python3.6/site-packages/pygments/cmdline.py is passed to ctags. The python parser of ctags caputres imported modules. ctags main part runs the python parser on the captured modules recursively when -S option is given. (To make the output smaller, only F kind entries are listed here.)

Combining multi pass parsing feature with this sematic recursion will be quite powerful; ctags can run C parser on a C source file with knowledges about all macro definitions used in the C source file.

masatake commented 3 years ago

An interesting question related to this issue https://stackoverflow.com/questions/65867522/create-tags-for-files-using-ctags

AmaiKinono commented 3 years ago

An interesting question related to this issue https://stackoverflow.com/questions/65867522/create-tags-for-files-using-ctags

It is an interesting one. I think at least it can be solved in 3 ways:

Let the client tool recognize the user's intend and transform chapters/chapter1 into chapter1.tex for matching. I think this is easy to hack if the client tool they use doesn't utilize readtags.
Let the client tool don't match by name, but by the input field, and filter those have file/F kind. This is easy with readtags.
Let ctags generate reference tags, then the client tool needs to make sure it grabs chapters/chapter1 as the symbol.

masatake commented 3 years ago

ctags itself also must have an ability to solve file name from given language object name when implementing multi-pass parsing. See if (strcmp(suffix, ".py") != 0) in https://github.com/universal-ctags/ctags/pull/2741/commits/4aef85d631c000457b389c0d3c81c11d3ce6a2f1#diff-40f2d292b59e7d5e66dc889a6f66ba3bab41657faacd15b9f87f6b6ed848dfc3R1097 .

Let ctags generate reference tags, then the client tool needs to make sure it grabs chapters/chapter1 as the symbol.

It is already done in ctags side!

[jet@living]~/var/ctags-new% ./ctags --extras=+r --fields=+rE -o - /tmp/foo.tex 
chapters/chapter1   /tmp/foo.tex    /^\\include{chapters\/chapter1}$/;" i   roles:included  extras:reference

universal-ctags / ctags

Making reference tags for external entities instead of definitions tags #2428