Cannot parse long lines of stdout: bufio.Scanner: token too long

[Spiral90210]

Hi,

We have a test which outputs some diagnostic information to stdout, which includes a json object with html in a field (incidentally, have had the same thing happen in our application code!).

The issue is the scanner in gotest.go created in func (p *Parser) Parse(r io.Reader) (gtr.Report, error) uses an underlying buffer with a max size of 64KB. There's 2 fixes to this I know of - switch to using bufio.NewReader with r.ReadLine() which makes things a little more complicated, or a handy hack to change the max capaciry of the scanners underlying buffer:

s := bufio.NewScanner(r)
maxCapacity := 4 * 1024*1024
buf := make([]byte, maxCapacity)
s.Buffer(buf, maxCapacity)

The correct fix is the former, but then you need to deal with putting lines together in code, wheras I have a hard time imagining a single line of text being over 4MB (although of course it is possible). In my case, a single MB was plenty large enough.

[jstemmer]

Thanks for reporting this issue and suggesting potential ways to fix it.

Whatever buffer size we choose to use, it could always be exceeded. So it sounds ReadLine() is the way to go, then we can at least drop or truncate any line that's too long and just continue parsing the rest rather than failing completely.

[Spiral90210]

Thanks!