benbjohnson / megajson

A JSON parser generator for high performance encoding and decoding in Go.
MIT License
467 stars 36 forks source link

Can't Unscan strings #13

Closed aybabtme closed 10 years ago

aybabtme commented 10 years ago

Strings are tokenized as TSTRING:

const (
    TSTRING = iota
    TNUMBER
    TTRUE
    TFALSE
    TNULL
    TLBRACE
    TRBRACE
    TLBRACKET
    TRBRACKET
    TCOLON
    TCOMMA
)

https://github.com/benbjohnson/megajson/blob/master/scanner/token.go#L4

The integer value of TSTRING is 0:

tok := scanner.TSTRING
fmt.Printf("%s=%d\n", scanner.TokenName(tok), tok)
// Output:
// string=0

When you Unscan a string token, the value s.tmp.tok becomes 0. However, in Scan:

if s.tmp.tok != 0 { // will never Scan an Unscanned TSTRING
    tok, b := s.tmp.tok, s.tmp.b
    s.tmp.tok, s.tmp.b = 0, nil
    return tok, b, nil
}

https://github.com/benbjohnson/megajson/blob/master/scanner/scanner.go#L117