Fix the toggling & add an optional faster C implemenation of tags filtering

[roflcopter4]

Hi again. I'm happy you merged the last pull request. Rather than reply there, I decided I'd fix the toggling thing and make one more little pull request instead. After rereading a bit of the python code I decided out of the blue that it would probably not be too difficult to rewrite some of the slower bits in C. Fast forward a few days and I have that working.

Assuming you're not completely sick of me at this point, it might be interesting to include this. I'm not actually sure if it's a great idea, given that C code needs to be built, but I tested it on a few monster files and it's often 5 or so times faster than the python. The largest file in the perl source tree takes on average more than 17 seconds to process with the python code on my computer, whereas the C code takes about 3-4 seconds.

Either way though, I did fix the issue with the toggling, and even if you do merge the C stuff it is entirely optional and the old code will run without any problems if not enabled. Building it should be easy too. It's autoconf'd (amusingly overkill for a program with 4 small source files) and I wrote a tiny Makefile wrapper that runs configure, make, make install, and make distclean in one go to make things easier for integrating with a vim plugin manager.

[c0r73x]

Haha sick of you? Absolutly not, this is awesome! :grinning:

I will test this today at work and merge it if i don't find any serious issues :smile:

Edit: Omg it's fast! :open_mouth:

[c0r73x]

I've tested this and I really really want to merge it :smile: but maybe we should write some note in the readme about compiling the c code.

[roflcopter4]

I'm glad to see that you like it. And that it actually works too. You're definitely right about the readme. I could easily update it for dealing with unix-ish systems - just type make - but as I discovered yesterday it's a bit of a nightmare with windows.

After some hackery I got it to compile in microsoft's visual c compiler (what on earth does that name even mean?) but it involved copypasting the mingw64 pcre2 binary libraries into the project tree because I couldn't find any reachable way to install them natively. For whatever reason the mingw version of gcc compiles a binary that is slower than the python (?!?) whereas microsoft's compiler is just as fast as gcc on linux. Seems kinda suspicious actually.

Not sure what to do here. I might end up just ignoring windows users. The python code still works after all. The only other thing I could think of is to include a whole visual studio project solution thingy and throw the dll binaries in with it (yuck). They're under the BSD license, so it wouldn't be too illegal, but it is a bit dubious.

I hate windows.

[c0r73x]

Anything on Windows is a nightmare imo. especially compiling code. I would say that it's better to let them use the python version and if they want to mess with it they can compile and optimize it for their OS.

[roflcopter4]

I agree with you there. Windows is just baffling. In any case, I did skim through enough of the cmake manual to convert the build system to it and it seems to mostly work. Not as robust as autotools imo but there are 4 sources files here, so who cares. Plus this way Github doesn't report that half the project is composed of makefiles like it does with autotools. Smaller too.

I updated the readme. Let me know if you think it's too long. I spent ages bashing my face against Windows trying to compile the thing and finally got it to work, so I really felt like explaining how. It might even be possible to just add that compressed binary to the releases section thingy if you wanted. It would have to include libpcre2, but it's BSD licensed, so that wouldn't be too difficult. In any case up to you. Maybe not worth the bother.

I also fixed up a few things. I noticed that it wasn't actually recognizing C++ code at all. Turns out python was giving the C binary the string C\\+\\+, which was definitely not what I expected. Now it should not only actually recognize C++ but will pretty much treat it and C as the same language as far as tags are concerned.

[roflcopter4]

I threw in what is (probably?) the last change I can really think to make right now, which is one I actually wanted to do for a while. The tag files can eventually take up an obnoxious amount of space, and they are extremely repetitive. Perfect candidates for compression. So I went ahead and did that.

The tag files are just compressed with gzip. It's fast and more than good enough for something this easy to compress. Using bzip2 or xz wouldn't be worth the extra time. More importantly though, zlib includes the magic macro gzgetc that works just like fgetc (somehow?), so supporting gzip from the C code took approximately 2 minutes. Then I spent several hours trying to work out how to properly deal with potentially malformed utf8 code in python.

Things should work now, though in retrospect this change really belongs in a separate pull request. The most notable problem here is that for some reason neovim can't read gzipped tag files. It wants them in plaintext or not at all. So I had to find a way to decompress them to a temporary file and add that to the &tags setting so that the ordinary tag stuff would work. That is probably buggy.

[c0r73x]

Sorry for the late response, I havn't been at a computer in a few days.

Ah cool, I will test it out today and merge it if everything works as expected :smiley:

[roflcopter4]

Awesome. Has everything been working? I tweaked my own copy a bit more (honestly the semester is over and I'm just messing around) so I'm not 100% sure if the current main branch is fully working.

And I see you added me to the project? Thanks!

[c0r73x]

Yes it work's perfectly, at least in C++, haven't tried any other languages so far.

Yeah I thought it would be easier, then you can push your changes to the dev branch instead of making huge pull requests :laughing: