Expose sdt elements as *token.Token with `$T0` etc.

[kfsone]

My code routinely makes use of raw tokens, which adds a LOT of boiler plate in translating these tokens from Attrib to *token.Token; either the BNF becomes a hideous mess of casts, or many of the most trivial functions are cluttered with them.

This change:

• compiles the sdt regex once, on startup, rather than per action,
• streamlines/simplifies code by using ReplaceAllString, (* if ReplaceAllString is more expensive than prior code, the one-time expression compile pays for it)
• allows $T0, ... $T9999 etc as producing X[n].(*token.Token),
• uses example/astx to demonstrate/test,
• updates the documentation to reflect,

[kfsone]

The build-failure appears to be a conflict between gofmt and goimports where golangci-lint is agreeing with gofmt:

-import "github.com/goccmack/gocc/example/astx/ast"
-import "github.com/goccmack/gocc/example/astx/token"
+import (
+       "github.com/goccmack/gocc/example/astx/ast"
+       "github.com/goccmack/gocc/example/astx/token"
+)

[kfsone]

Making goimports part of regenerate in the Makefile resolved the issue. Also, I removed/ignored go.sum, it's not usually supposed to be included, but was there a reason for it to be? (and main.go.orig?)

[awalterschulze]

Here is the same diff issue as I found in the other pull request. Why does the whole file seem to be changed, if only a few lines changed?

Btw it is fine to keep the || echo change as part of this pull request. We just need the diff to fixed.

[kfsone]

@awalterschulze Re the question about "token" containing everything but the filename, from past experience it doesn't feel like necessarily the correct place to have the filename, per se, rather the lexer or parser. (I tend to go even more spartan on tokens: https://github.com/kfsone/parsing/blob/main/token.go, https://github.com/kfsone/parsing/blob/main/symbol.go, and finally exposing the origin in the lexer, thus making it visible to the parser: https://github.com/kfsone/parsing/blob/main/lexer.go; that's just a noddy learn-myself-some-go project I hacked up last month)

I'm thinking gocc might be a great basis to explore my "structured rules" ideas... Just pasted them as gists: https://gist.github.com/kfsone/bbf61478bf2df3c4b02d08b3524111cb and https://gist.github.com/kfsone/9a3c9737fc571bb603252816c330f0b3 (fleshed out sample)

[awalterschulze]

Here is the same diff issue as I found in the other pull request. Why does the whole file seem to be changed, if only a few lines changed?

Btw it is fine to keep the || echo change as part of this pull request. We just need the diff to fixed.

Just want to say, it seems this has been fixed, so ready for someone else to review again.

[awalterschulze]

@awalterschulze Re the question about "token" containing everything but the filename, from past experience it doesn't feel like necessarily the correct place to have the filename, per se, rather the lexer or parser. (I tend to go even more spartan on tokens: https://github.com/kfsone/parsing/blob/main/token.go, https://github.com/kfsone/parsing/blob/main/symbol.go, and finally exposing the origin in the lexer, thus making it visible to the parser: https://github.com/kfsone/parsing/blob/main/lexer.go; that's just a noddy learn-myself-some-go project I hacked up last month)

I'm thinking gocc might be a great basis to explore my "structured rules" ideas... Just pasted them as gists: https://gist.github.com/kfsone/bbf61478bf2df3c4b02d08b3524111cb and https://gist.github.com/kfsone/9a3c9737fc571bb603252816c330f0b3 (fleshed out sample)

Maybe this is worth creating a new issue for or adding the to the more context for a token issue?

[kfsone]

Rebased.