search

zendesk / ultragrep

the grep that greps the hardest.
Apache License 2.0
29 stars 4 forks source link

Allow configuring a sane upper limit on IO threads #50

Closed Nebopolis closed 7 years ago

Nebopolis commented 7 years ago

Description

This PR adds a new configuration option concurrency_limit which forces the batches of files to be read and then closed in smaller chunks, rather than keeping a process open for each file found in the glob search. For servers with a large number of logfiles, even at low IO and CPU priority ultragrep could still cause IO starvation due to all the processes opening files at once.

Tasks/Steps to Merge

References

/cc @zendesk/sustaining

Risks

Log results may be returned in a slightly different order, but all the same results will still be present.

Nebopolis commented 7 years ago

/cc @zendesk/sustaining @zendesk/znoc

pdeuter commented 7 years ago

🐔