search

magnusbaeck / logstash-filter-verifier

Apache License 2.0
191 stars 27 forks source link

Failed test case with a ~70Kb input line, reading standard output: `bufio.Scanner: token too long` #169

Closed apple-corps closed 2 years ago

apple-corps commented 2 years ago

I have an input line about ~70Kb long. Perhaps a little longer after escaping quotes for the JSON syntax. I fail my test cases but the output seems a little odd.

Then looking at the debugging output from LFV2 beta:

reading standard output: bufio.Scanner: token too long
stdout scanner closed unexpectedly
state change: "running_test" -> "unknown" by "crash"
exit stdout scanner
failed to wait for Logstash results: shutdown while waiting for state: ready_for_test
broadcast shutdown for waitForState
shutdown log reader

Then I think the bufio Scanner is likely in https://github.com/magnusbaeck/logstash-filter-verifier/blob/master/internal/logstash/eventreader.go or https://github.com/magnusbaeck/logstash-filter-verifier/blob/master/internal/daemon/instance/logstash/processors.go

I'm not a go programmer but can the token size be made configurable for users who want to test longer log sizes which might be closer to their use?

breml commented 2 years ago

Here we hit the default bufio.MaxScanTokenSize defined in the Go Stdlib, which is defined as 64KB.

It is possible to increase the size of this buffer, but is it really necessary? In order to keep the tests maintainable, I feel like having test cases with 70KB of input is no longer maintainable. Wouldn't it be possible to test the correct behavior of the Logstash config with smaller inputs?