Token: Literal string

[kfsone]

token.Token inlines Pos rather than having it as a member:

type Token struct {
  Type
  Lit []byte
  Pos
}

This means that in the absence of it's own String() method, token.String() is actually Pos.String(), which means that printing a token actually prints its position?

  t := &token.Token{ Type: 100, Lit: []byte("fubar"), Pos: token.Pos{ 0, 0, 0 } }
  fmt.Printf("%s %s\n", t, t.String())

outputs:

Pos(offset=0, line=0, column=0) Pos(offset=0, line=0, column=0)

but what the user probably expected would have been

fubar fubar

This could be fixed:

• Give Token it's own String() so that if you want the pos you use token.Pos.String(); could break existing code (if someone is relying on it to print Pos(offset=...)?)
• Add a Literal() function to Token:

  func (t *Token) Literal() string { return string(t.Lit) }

I've only been using go for a year or two, so I thought the go style was to put inheritances at the top and totally missed Pos vs Pos Pos.

[kfsone]

I guess this is what IDValue is for :)