search

zendesk / ultragrep

the grep that greps the hardest.
Apache License 2.0
29 stars 4 forks source link

Too many threads, CPU use #28

Open babbottscott opened 10 years ago

babbottscott commented 10 years ago

I seem to remember a configurable thread limit per ultragrep process. This appears to be no longer the case. cpu use is too high on logs servers with a thread per log file searched. There needs to be some cap.

osheroff commented 10 years ago

maybe tricky to pull off given the architecture of a single C process per log file grepped. We could try rate limiting by forcing ug_guts to block writing to its pipe, but to do it "right" would probably mean a significant re-architecture, making ug_guts a multi-threaded process.

Nothing we could do with nice-levels?

babbottscott commented 10 years ago

It's already renicing to +19. There was a suggestion to use taskset to add affinity to a subset of cpus. Perhaps that would be a cleaner implementation?

osheroff commented 10 years ago

yeah, that could be fine. How many cpus need we reserve?

babbottscott commented 10 years ago

First take I'd say 8 out of the 24 at most, but I'd love to see it configurable. Taskset takes strings like '1-4,6,8' for specifying affinity. If we could specify that string in the config, that would be great.

briser commented 7 years ago

dusts off thread Bump.