if GTAGSLABEL=pygments, htags doesn't create HTML files.

[nao16t]

If GTAGSLABEL=pygments, gtags works fine, but htags doesn't create HTML files properly. If GTAGSLABEL=default, both gtags and htags work as expected. Is it a current limitation?

export GTAGSCONF=/usr/local/share/gtags/gtags.conf export GTAGSLABEL=pygments gtags -vw --debug htags --suggest

[yoshizow]

What do you mean by "htags doesn't create HTML files properly"? If you mean that "MAINS section and DEFINITIONS section are empty", you need to install ctags.

global-pygments-plugin itself does not generate definition tags. It just generate symbol tags. So htags will not generate MAINS section and DEFINITIONS section if you only use global-pygments-plugin. If ctags is installed on your machine and you enable gtags to use it (with configure --with-exuberant-ctags=XXX option), global-pygments-plugin will use ctags to generate definition tags. In that case htags will successfully generate MAINS section and DEFINITIONS section.

[nao16t]

Sorry for my late reply. ctag is installed in my environment. Please take a look at the contents of HTML directory below in case of GTAGSLABEL=pygments. Several html files are missing comparing the case of GTAGSLABEL=default. Is it possible to show us how to use htags after gtags in README.md ?

nao16t@ubuntu2:~/work/src/global-6.3.2$ python --version Python 2.7.6 nao16t@ubuntu2:~/work/src/global-6.3.2$ pip list | grep Pygments Pygments (1.6) nao16t@ubuntu2:~/work/src/global-6.3.2$ type ctags ctags is /usr/bin/ctags nao16t@ubuntu2:~/work/src/global-6.3.2$ ctags --version Exuberant Ctags 5.9~svn20110310, Copyright (C) 1996-2009 Darren Hiebert Compiled: Oct 7 2014, 13:52:37 Addresses: dhiebert@users.sourceforge.net, http://ctags.sourceforge.net Optional compiled features: +wildcards, +regex nao16t@ubuntu2:~/work/src/global-6.3.2$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

nao16t@ubuntu2:~/work/src/global-6.3.2$ export GTAGSCONF=/usr/local/share/gtags/gtags.conf nao16t@ubuntu2:~/work/src/global-6.3.2$ export GTAGSLABEL=default nao16t@ubuntu2:~/work/src/global-6.3.2$ gtags -vw --debug nao16t@ubuntu2:~/work/src/global-6.3.2$ htags --suggest nao16t@ubuntu2:~/work/src/global-6.3.2$ ll HTML/ total 268 drwxrwxr-x 11 nao16t nao16t 4096 Nov 9 22:00 ./ drwxr-xr-x 23 nao16t nao16t 4096 Nov 9 22:00 ../ -rw-rw-r-- 1 nao16t nao16t 0 Nov 9 22:00 compress drwxrwxr-x 2 nao16t nao16t 20480 Nov 9 22:00 D/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:00 defines/ -rw-rw-r-- 1 nao16t nao16t 3368 Nov 9 22:00 defines.html -rw-rw-r-- 1 nao16t nao16t 12616 Nov 9 22:00 FILEMAP drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:00 files/ -rw-rw-r-- 1 nao16t nao16t 10734 Nov 9 22:00 files.html -rw-rw-r-- 1 nao16t nao16t 1804 Nov 9 22:00 help.html drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:00 I/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:00 icons/ -rw-rw-r-- 1 nao16t nao16t 15023 Nov 9 22:00 index.html drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:00 J/ -rw-rw-r-- 1 nao16t nao16t 15023 Nov 9 22:00 mains.html drwxrwxr-x 2 nao16t nao16t 65536 Nov 9 22:00 R/ -rw-r----- 1 nao16t nao16t 579 Nov 9 22:00 rebuild.sh drwxrwxr-x 2 nao16t nao16t 12288 Nov 9 22:00 S/ -rwxrwxr-x 1 nao16t nao16t 2049 Nov 9 22:00 style.css* drwxrwxr-x 2 nao16t nao16t 69632 Nov 9 22:00 Y/

nao16t@ubuntu2:~/work/src/global-6.3.2$ rm -r HTML/ nao16t@ubuntu2:~/work/src/global-6.3.2$ rm G* nao16t@ubuntu2:~/work/src/global-6.3.2$ export GTAGSLABEL=pygments nao16t@ubuntu2:~/work/src/global-6.3.2$ gtags -vw --debug nao16t@ubuntu2:~/work/src/global-6.3.2$ htags --suggest nao16t@ubuntu2:~/work/src/global-6.3.2$ ll HTML/ total 128 drwxrwxr-x 11 nao16t nao16t 4096 Nov 9 22:01 ./ drwxr-xr-x 23 nao16t nao16t 4096 Nov 9 22:01 ../ -rw-rw-r-- 1 nao16t nao16t 0 Nov 9 22:01 compress drwxrwxr-x 2 nao16t nao16t 20480 Nov 9 22:01 D/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 defines/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 files/ -rw-rw-r-- 1 nao16t nao16t 1804 Nov 9 22:01 help.html drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 I/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 icons/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 J/ drwxrwxr-x 2 nao16t nao16t 69632 Nov 9 22:01 R/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 S/ drwxrwxr-x 2 nao16t nao16t 4096 Nov 9 22:01 Y/

[yoshizow]

Thank you for the detailed report.

I tried with exactly same version set of software, but the issue was not reproduced. I tried generating tags on the GLOBAL's source code itself. Probably the issue depends on the content of the source code which GLOBAL (and plugin) processes.

From your report, it seems that htags exited during in the middle of its processing. Seems that some part of files are not generated because htags exited halfway.

Could you provide me the log of htags by running like this?

htags --suggest > htags.log 2>&1

By checking the log of htags, we can see when (and maybe why) htags exit.

[nao16t]

Hi, I took the log, and an error happened as you expected.

htags: terminated abnormally ''/usr/local/bin/global' -xs --result=ctags-xid --encode-path=" " --nofilter=path ".*"' (errno = 0).

But I couldn't figure out how to upload the htags.log, so I paste the last part. When I executed htags in GTAGSLABEL=pygments environment using G* files which gtags created in GTAGSLABEL=default, htags worked without errors. So I assume G* files which gtags created in GTAGSLABEL=pygments environment would have something to do with this problem.

If you need the whole log file and G* files, please let me know how to upload. Thanks.

---

[3277] adding reference yytnamerr [3278] adding reference yytoknum [3279] adding reference yytype_int16 [3280] adding reference yytype_int8 [3281] adding reference yytype_uint16 [3282] adding reference yytype_uint8 [3283] adding reference yyunput [3284] adding reference yyvs_alloc [1] adding symbol " [2] adding symbol "@cEnd:@cvim:sw [3] adding symbol "@ctime [4] adding symbol # [5] adding symbol $ [6] adding symbol $# [7] adding symbol $$ [8] adding symbol $1 global: Abbrev character must be a lower alphabetic character. (^@) [9] adding symbol $2 [10] adding symbol $4 [11] adding symbol $? [12] adding symbol $@ htags: terminated abnormally ''/usr/local/bin/global' -xs --result=ctags-xid --encode-path=" " --nofilter=path ".*"' (errno = 0).

[yoshizow]

Thanks. This log was helpful, but I couldn't figure out the detail.

Could you send me the whole log file and G* files via email? : yoshizow@turtlewalk.org Thank you for your cooperation!

[nao16t]

I sent the G* files and the log file by email. Please find them. Thanks.

[yoshizow]

By investigating your G* files, I succeeded to reproduce the issue.

gtags seems to have some problem with tag names which are composed of symbol characters like "$@". To work around this, you can set environment variable GTAGSPYGMENTSOPTS before running gtags:

$ export GTAGSPYGMENTSOPTS=strippunctuation
$ gtags -vw --debug

GTAGSPYGMENTSOPTS=strippunctuation option directs pygments plugin to strip symbolic characters from tag names.

I will investigate gtags implementation further and will make a patch for gtags. Thanks.

[nao16t]

GTAGSPYGMENTSOPTS=strippunctuation option stopped the error, thanks! But does this option cause not to create a tag for any token which contains symbolic character?

[yoshizow]

GTAGSPYGMENTSOPTS=strippunctuation option strips symbolic characters from tokens. If a token does not contain any character except symbolic character, the entire token is not recorded. e.g.

gsub! -> gsub
$foo -> foo
$@ -> (entire token is not recorded)

[nao16t]

I got it. thanks!

[yoshizow]

I have sent a patch for gtags. http://lists.gnu.org/archive/html/bug-global/2014-11/msg00034.html

[nao16t]

yoshizow,

It worked great without setting GTAGSPYGMENTSOPTS after applying the patch into global-6.3.2. Thanks!

One feature I expected is that gtags creates tag for python decorator which symbol starts with '@'. But global-6.3.2 didn't create it in my environment. Is it my environment setting problem, or just unsupported feature at this moment?

Best regards

[yoshizow]

This is because ctags does not include '@' for the symbol.

$ cat test.py
def deco(func):
    return func

@deco
def test():
    pass

$ gtags
$ gtags -d GTAGS 
 __.COMPNAME     __.COMPNAME
 __.COMPRESS     __.COMPRESS ddefine ttypedef
 __.VERSION  __.VERSION 6
deco    1 @n 1 def @n(func):
test    1 @n 6 def @n():

$ gtags -d GRTAGS
 __.COMPACT  __.COMPACT
 __.COMPLINE     __.COMPLINE
 __.COMPNAME     __.COMPNAME
 __.VERSION  __.VERSION 6
@deco   1 @@deco 5
func    1 @n 1

Though pygments plugin generates symbol '@deco' in GRTAGS, ctags generates symbol 'deco' in GTAGS. In htags output, symbols from GTAGS are listed in the DEFINITIONS section. Ctags and pygments plugin do not necessarily generate exactly same symbols. This is a limitation of pygments plugin.

[nao16t]

Aha, now I understood the explanation of "Customizing Behavior" in README.md. So in case of python as well as ruby, "export GTAGSPYGMENTSOPTS=strippunctuation" is necessary, I think. Thanks.