search

Clever / optimus

Concurrently extract, transform, and load tables of data in Go
Apache License 2.0
34 stars 7 forks source link

Custom scanner to support larger token sizes #24

Closed kvigen closed 9 years ago

kvigen commented 9 years ago

Before this change we would hit the "bufio.Scanner: token too long" error when processing really long json structures. To address this we created our own version of bufio.scanner that supported longer tokens. We did this by simply copying the original bufio code and changing the MaxScanTokenSize variable.

We changed the limit to 16MB to support deserializing Mongo documents written to JSON (Mongo's current doc size limit is 16MB)

kvigen commented 9 years ago

@azylman we ended up hitting this issue. Thoughts on whether we should make a separate repo just for our own version of scanner? Or is putting it in its each repo easy enough?

azylman commented 9 years ago

I think we should eventually make our own repo since this is ~5th time we've done this. It's fine to put it in optimus for now, but we should consolidate at some point.

azylman commented 9 years ago

lgtm

jiancheung commented 9 years ago

@kv I know nothing about this code but is this relevant? https://groups.google.com/d/msg/golang-nuts/bgK1uCgwkxg/Nn2B8hNn-agJ

kvigen commented 9 years ago

@jiancheung interesting, yes that is relevant. I think for now I'm just going to do this, but I'll look into what it would take to change our code to use ReadLine.