search

blue-yonder / bonfire

A CLI Graylog Client with Follow Mode
BSD 3-Clause "New" or "Revised" License
72 stars 26 forks source link

Every time I use `-f` I get the `RuntimeError: Query returns more than 10000 log entries. Use offsets to query in chunks.` #27

Closed dlamblin closed 4 years ago

dlamblin commented 6 years ago

This RuntimeError occurs not for the first printed output, but the second after a 1s wait.

Obviously its generated at this line. https://github.com/blue-yonder/bonfire/blob/d0af9ca10394f366cfa3c60f0741f1f0918011c2/bonfire/graylog_api.py#L141

Given that a -f can't specify a limit, what should I be doing to use it correctly?

valentin-krasontovitsch commented 5 years ago

yeah the follow mode is a bit problematic - i think right now it's just not taking into consideration that there might be more than 10000 results in one second!?

i would love to change the way it queries in follow mode (so instead of querying all matching logs from the next second interval, query say the next 100 matching log), but i'm not sure that it's possible.

in the meantime, a solution for querying huge dataloads in followmode might be to decrease the interval from one second to a tenth of a second, for instance?

On the other hand, is it really reasonable to query in follow-mode when you have more than 10000 logs per second? that's gotta be hard to follow on screen... perhaps you could adjust your query to get the results you really want to see?

valentin-krasontovitsch commented 4 years ago

perhaps this is an x y problem - closing for now, feel free to reopen with a detailed use-case that does not permit a work-around by just using the tool differently.