search

Genivia / ugrep

NEW ugrep 7.0: a more powerful, ultra fast, user-friendly, compatible grep. Includes a TUI, Google-like Boolean search with AND/OR/NOT, fuzzy search, hexdumps, searches (nested) archives (zip, 7z, tar, pax, cpio), compressed files (gz, Z, bz2, lzma, xz, lz4, zstd, brotli), pdfs, docs, and more
https://ugrep.com
BSD 3-Clause "New" or "Revised" License
2.64k stars 112 forks source link

Q mode usability #282

Closed nkh closed 1 year ago

nkh commented 1 year ago

I installed ug on my machine few years ago and I used it 1 minute total.

I first installed it for speed, I sometimes have some large amount of data and need to find specific files, yes it's because I forget where they are, not that I get new data all the time ;)

Speed is indeed very good but being a grep and rg user I've never really missed the speed and waiting 10% of the time more doesn't make a difference in my case.

the second interest was interactive queries, coming from fzf I very quickly stopped trying ug:

and the feeling that search results are not kept but everything started again, I understand the effect on memory usage even though I wonder if the search itself wouldn't be faster if the query is analyzed and the next search is found to be a subset of the previous results.

of course none of those are not fixable with a few options but I believe that because of those points there is very little incentive for people to leave FZF which is a pity because I've seen a lot of people wanting to do non fuzzy searches and sometime to go from one to the other, something I believe ug is better fitted for

I hope these points can be of help and even if they sound negative my intent was to give honest feedback and maybe a few ideas for improvement

genivia-inc commented 1 year ago

the second interest was interactive queries, coming from fzf I very quickly stopped trying ug:

  • the low rate of refresh even with Q set to 1
  • the unsightly [end] block
  • the colors
  • the confirmation for quitting
  • the [searching start] flickering

A few notes since you may have missed it in the man page and the README and help page that tells you what to do:

Ugrep is very easy to configure if you spent a few minutes reading the instructions.

EDIT: the reason why ugrep start/stops the search is that it is designed to be limitless. What does that mean? Files being searched are not stored in RAM for one. This way, ugrep in the TUI can search several GB or even the entire file system relatively quickly without keeping all the files in RAM. It also temporarily stops searching if the results aren't displayed to keep CPU cost low, until scrolling down to view the results. However, if the search pattern doesn't match many files, then ugrep obviously runs hot to search until it finds matches to display. The use cases for ugrep are not dissimilar to FZF, but the intent to use it as an interactive grep and its search capabilities differ.

genivia-inc commented 1 year ago

I first installed it for speed, I sometimes have some large amount of data and need to find specific files, yes it's because I forget where they are, not that I get new data all the time ;)

Speed is indeed very good but being a grep and rg user I've never really missed the speed and waiting 10% of the time more doesn't make a difference in my case.

Well, the new ugrep with benchmarks show that ugrep is almost always significantly faster than rg and grep. Some patterns take seconds to search with rg and grep when ugrep only takes a few tens of a second.

There are still opportunities to speed up ugrep so ugrep will continue to get faster.

Development on ugrep started in earnest only two years ago (it was a prototype RE/flex grep example before that) with new optimizations and new features. Now is the time to converge to an even faster and more capable ugrep.

nkh commented 1 year ago

I wouldn't call FZF light weight, nor Rg slow, since I'm trying to use ug more I did use it this afternoon and for the use case I had I don't need benchmarks to know that ug is slower, of course it's because the setting are different, they find a completely different number of files, so more tweaking is necessary before I have a first hand well setup ug. Something I'm looking forward to.

You certainly have reasons for not interrupting the running processes but in the human waiting for feedback realm, the delay is felt, and I run a linux machine, plenty ram and fast ssd. keeping files in ram doesn't slow down search, or please explain to me why as I may be wrong and I'm curious, on the other hand most searches are iterative which means less matches for each search which means fewer file processing. But we all have good reasons for architectural decision I just want to be a ball plank.

I'm happy to read that there are options to mitigate some of my concerns about the UI, but don't take my word for it, put someone you trust in front of FZF and ug -Q1 and hear what they say about the UI look and feel. It's of course normal as FZF has had years to polish things up, with the same amount of effort I have no doubt they would be both as pleasant to use. And I need to spend more time in the docs and try all the tweaks.

It's great to hear that there are more opportunities for speed improvement even if it's not my use case, it will help other. if ug is just around two years old then I must have caught one of the first release, I'm happy the project is a success.

Looking familiar and looking good is not the same, and I have the less colors changed (will do the same with ug) but less doesn't have [end] blink 10 times a second.

thank you for answering or commenting the points I listed

Once I'm more familiar with ug I'll also write about the things I like, I can tell you that --format is going to be one of them.

Thank you for the answers.

genivia-inc commented 1 year ago

I wouldn't call FZF light weight, nor Rg slow, since I'm trying to use ug more I did use it this afternoon and for the use case I had I don't need benchmarks to know that ug is slower, of course it's because the setting are different, they find a completely different number of files, so more tweaking is necessary before I have a first hand well setup ug. Something I'm looking forward to.

Lightweight because of low thread and CPU utilization to search. Ugrep uses a thread pool to speed up recursive searching and is more CPU intensive for those tasks. My comment has nothing to do with the size of the tool or features. Hope this clarifies that.

Also, there will always be some patterns for which one or the other tool runs faster. Different algorithms give different performance results. There are some things that I know how and where to optimize in ugrep. It's not difficult, just making sure there are no bugs. Ripgrep has bugs actually, likely because of optimizations that create many logic flows that aren't all tested and verified. That is a nightmare (for me, at least).

I'm happy to read that there are options to mitigate some of my concerns about the UI, but don't take my word for it, put someone you trust in front of FZF and ug -Q1 and hear what they say about the UI look and feel. It's of course normal as FZF has had years to polish things up, with the same amount of effort I have no doubt they would be both as pleasant to use. And I need to spend more time in the docs and try all the tweaks.

In order to address concerns, I need to know what machine is used. The windows terminal is not a good one and ugrep's response will feel slower.

Also, if you restart a search by typing, then the previous search must be cancelled. If the previous search is searching a huge compressed file then this cannot be terminated immediately as this will cause problems with the compression libraries such as memory leaks if terminated without proper close. That is why the decompression has to complete first, which will increase the response time for the next search.

I have yet to hear from someone that this can be done more cleverly. Running forked processes can be killed. But that is a design that has a lot of overhead. And no, IPC with a forked process is not a better solution, because IPC is not sufficient to control the forked process to terminate and clean up etc. Threading is faster and gives more control.

Things are not as trivial as one might believe.

Looking familiar and looking good is not the same, and I have the less colors changed (will do the same with ug) but less doesn't have [end] blink 10 times a second.

What machine and terminal are you using? I don't see this at all on Linux-based machines.

thank you for answering or commenting the points I listed

  • the low rate of refresh even with Q set to 1 -> architectural

No. This is fundamental. Ugrep is a "classic" grep tool but with a TUI (and more). Any TUI must terminate and rerun a search with 8 threads in the background. This adds latency. Slowing down search to improve the TUI response time is not a good option either. If you want to limit search overhead, just use option -J1 to run only one search thread. That can speed up the TUI response time.

  • the unsightly [end] block -> option, not sure which one in the GREP_COLORS

Not in GREP_COLORS. It's just the same as more and less show at the end. Without it, it is not clear when the search ended. Believe me. I had an implementation without the [end] but had to scroll way past the end to feel confident that the search ended. Consider the implications for the user when there is no [end].

I don't see flickering on Linux-like machines and nobody else has complained about that either. Windows has this problem due to their terrible terminal designs and the fact that only polling 10 times a second allows responses to window events to be registered, instead of Linux events that do not need polling. So again, what machine/OS are you using?

The fact that the ugrep TUI window is responsive to mouse scrolling and resize is important too.

  • the colors -> option
  • the confirmation for quitting -> option, an a good one! the [searching start] flickering -> not answered

See above.

genivia-inc commented 1 year ago

Still looking for responses to my questions (inline) to answer your questions and a few more to understand your concerns.

nkh commented 1 year ago
genivia-inc commented 1 year ago

Ah, finally... thanks for the details.

Test results to enter a search string, change options, move cursor, scroll screen:

Not entirely surprising, because there is no terminal polling performed by ugrep on *nix systems. Flickering is a known issue with Windows ugrep.exe because of polling to allow reactive window resizing.