aswild
commented
3 years ago Hey, thanks for the shout-out. One word of warning - most of my open PRs on this repo are pretty old and in most cases I have better versions of them in my own fork. I sorta lost interest in pushing PRs here due to the general trend of only small tweaks being accepted rather than larger features/fixes/refactors.
Anyway, if you want to keep rolling with PCRE2 in your fork, check out https://github.com/aswild/the_silver_searcher/commit/bbd3ee46113e3e7f13a75ba65fa35da06b3fe2d1 where I removed PCRE1 support and got considerable speedup by reducing heap thrashing on every call to pcre2_exec.
As far as searching big files goes, I think that's a fundamental/hard problem in ag. Except when searching stdin in stream mode, the whole file to be searched has to be buffered in memory, either with mmap or by reading it into the heap. There's even a check that the file length isn't greater than INT_MAX due to PCRE limitations*. Naturally, this leads to memory pressure and performance issues on some systems. This is further complicated by the way ag saves matches as pointers into the full file buffer, and since all matches are collected before stdout is mutex-locked, it means the whole buffer has to wait around in memory before any of its matches get printed to the console.
I don't mean to sound discouraging, these aren't insurmountable roadblocks, but they are the product of multiple interlocking facets of ag's design that would require significant effort to refactor.
* I just realized that this limitation isn't needed in PCRE2 and I should probably remove it in my fork.
gcflymoto
I really love "ag" and use it every day for work as a developer, but it has limitations when searching big files. I have put together my own fork with many PRs which haven't made it into this repo, and it gets a lot closer. It includes PCRE2 from #1035 but it still has trouble with big files. Any way I can sponsor somebody and how do I go about it?
Much thanks to @aswild @napcode @afredd @mwagnell @jtamboli @thebaptiste for providing the PRs!!
@aswild Add support for libpcre2 (compiles and works) #1035 @aswild fix compile warnings on GCC10 #1398 @aswild search_file: check return code of read for failure #1260 @napcode Fixes overflows for large and compressed files #1221 @afredd Rework is_binary() #204 @mwagnell Segmentation fault when searching an xlsx file with -z option #1349 @jtamboli Include file names & line numbers when searching paths, even if stream is present #1433 @thebaptiste To be able to build with lzma.h missing or zlib.h missing or both mis… #1353