incorrect parsing of exuberant ctags tags file

[GoogleCodeExporter]

What steps will reproduce the problem?
1. cd into vim/src
2. Run ctags with case folding(ctags -R --sort=foldcase)
3. Open vim/src/main.c
4. Either run:
    :set tags=tags
   Or your vimrc will need to be set to load the tags file
5. Go to line 208, and try to jump to definition for mch_fopen  (CTRL-] is the 
specific command I'm using to try to jump)
       E432: Tags file not sorted: tags
       E426: tag not found: mch_fopen

What is the expected output? What do you see instead?

I'd expect to follow jump to the definition of mch_fopen.  I get the errors 
list above instead
      E432: Tags file not sorted: tags
      E426: tag not found: mch_fopen

What version of the product are you using? On what operating system?

    Centos6 up todate as of yesterday
    vim hg pull -u from a few days ago
    Features: huge
    Both python and ruby enabled and pointing to pythonbrew and rvm installations
        VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Oct 30 2012 16:11:40)
        Included patches: 1-712
   ctags version: 5.8

Please provide any additional information below.

If you look at the tags file you'll see a line at the top starting:
   !rm
Which due to foldcase has been sorted before the "!_TAG_XXXXX" lines.
This is valid tags file according to: http://ctags.sourceforge.net/FORMAT
Written by Bram and Darren Hiebert

Particularly:

    The names of the tags used in these lines may be suitably chosen to ensure
    that when sorted, they will always be located near the first lines of the tag
    file.  The use of "!_TAG_" is recommended.  Note that a rare tag like "!" 
    can sort to before these lines.  The program reading the tags file should be
    smart enough to skip over these tags.

Vim is failing to read far enough into the file and assumes that the file isn't 
sorted even though it is, which then leads down to further errors and not 
finding the tag that is there.  I looked a few weeks ago at tag.c I think 
TS_START is getting unset too soon (though it might need more work than that).  
Vim might need to readjust how it searches when it finds "!_TAG_XXXX" lines.  I 
didn't have the time to try coming up with a patch (or even to file a bug until 
today....)

If you don't set ctags to foldcase things will work (maybe not as fast a tags 
lookup when ignorecase is set).  I have a few other source trees I'm working 
with that have some other rare tags that get sorted above the "!_TAG_XXXX" set 
of lines, and seen similar problems.

Original issue reported on code.google.com by darren.c...@gmail.com on 2 Nov 2012 at 7:57

[GoogleCodeExporter]

I am not able to reproduce the problem. It works here without any problems.

Vim version 7.3.753 and 7.3.785
ctags:
Exuberant Ctags 5.9~svn20110310, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Apr 30 2012, 19:23:25
  Addresses: <dhiebert@users.sourceforge.net>, http://ctags.sourceforge.net
  Optional compiled features: +wildcards, +regex

Perhaps, you need a newer ctags version?

Original comment by chrisbr...@googlemail.com on 28 Jan 2013 at 9:46

[GoogleCodeExporter]

I can reproduce something similar. I prepare the file as specified by the OP 
and then instead of using normal-mode tag command I try to execute a 
command-line mode command:
:tag !
and after the exclamation point I use TAB to get possible completions. This 
results in E432 and - surprisingly - a list of completions.

I'm working under Cygwin with the following versions:
Vim 7.3.772
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert

Original comment by lech.lor...@gmail.com on 28 Jan 2013 at 11:33

[GoogleCodeExporter]

I still get an error, but only the second one.  Followed the same test as above 
but running binaries from the just build source trees.  Here are the details.

Built vim from source pulled down today (7.3.785) and ctags from svn source 
pulled today (svn rev 804)

I ran the vim binary from within the tree.  I ran without any vim runtime files 
installed.  I moved aside my .vim, .vimrc (though I get the same error even 
with my setup and no runtime files).  Default value for tags option used was: 
tags=./tags,tags

I ran ctags from the source tree I built.  The files have differences from each 
other, but there is still a !rm tag sorted to the top  and tag lines for 
mch_fopen exist (though there are differences in those lines).  Again ctags run 
with case folding: ctags -R --sort=foldcase

I still get an error, but only the second one:
     E426: tag not found: mch_fopen

If I run than vim compiled today with a tags file from the installed ctags 
(5.8) I get the same two errors in my original report.  So ctags version is 
having some affect.

I haven't looked at the source yet to see what is going wrong.  I wonder if the 
line ctags header line:
   !_TAG_PROGRAM_VERSION   Development
Is causing problems with my tests.

I use vim on a daily basis so I'm very willing to help.  Though I don't know 
when I'll have a chance to step through the vim code to see if I can find the 
problem.

Some further details on exactly what I am running.

I'm running on Mac OS 10.7.5

vim built source pulled down today first few lines of --version
    VIM - Vi IMproved 7.3 (2010 Aug 15, compiled Jan 28 2013 08:25:37)
    MacOS X (unix) version
    Included patches: 1-785

ctags built from source doesn't show 5.9, but development instead.  I built 
from trunk.   lines from --version
   Exuberant Ctags Development, Copyright (C) 1996-2009 Darren Hiebert
     Compiled: Jan 28 2013, 08:33:37
     Addresses: <dhiebert@users.sourceforge.net>, http://ctags.sourceforge.net
     Optional compiled features: +wildcards, +regex
Maybe that was because of how I ran autoheader, autoconf, and then just 
./configure --prefix=....  I didn't get the 5.9 version

Original comment by darren.c...@gmail.com on 28 Jan 2013 at 5:12

[GoogleCodeExporter]

Problem is that case-folding sorts a tag such as "!rm" before "!_TAG", but the 
check for the end of the header does not take that into account.  With the 
result that the header line indicating that the tags file is case-folded is not 
found

Fixed by patch 7.3.1202.

Original comment by brammool...@gmail.com on 15 Jun 2013 at 8:27

• Changed state: Fixed