Getting stateful context in an action or for a token

[kfsone]

Running a bunch of Parse()rs concurrently, is there a current mechanism or best-practice for providing parser or lexer context to ast functions? I'm ultimately wanting to be able to capture the file/line/column source of certain tokens so during AST validation I can advise the user on sources of conflict:

guide.chapter7.txt:42:11: name 'slartibartfast' is already used. see: guide.chapter6.txt:6:9: previous use

I don't see anything obvious - the regex for $ is [0-9], and ReduceFunc only takes one argument, X []Attrib.

[kfsone]

If there is not, I'll have a PR which adds a "UserContext interface{}" to parser.Parser and which, when the field is not nil, appends it to the Attrib list made available to actions.

File: Imports;

Imports
    : "import" identifier << ast.NewImport($1, $2) >>   /* $2 is the user context */
    | Imports "import" identifier << ast.AddImport($0, $2, $3) >> /* $3 is the user context */

[kfsone]

Instead of making it just the last of the attribs, I made it '$Context'. Aside from obvious things like filename visibility in ast functions (without using a global), I'm looking to use it to work around the end-of-sdt only implementation of actions to allow myself to do yacc-style inline type matching/validation of the dsl I'm parsing:

StructMember
    : FieldName TypeName "=" { startField($0, $1, globalContext); } Value { endField($3, globalContext); }
    | FieldName TypeName { startField($0, $1, globalContext); endField(nil, globalContext); }
    ;

where startField pushes the type onto a stack that ast functions in Value can use to tell the user if they're providing an array where an int is expected, etc.

// gocc
StructMember
    : FieldDef "=" Value << ast.EndField($0, $1, $Context) >>   // value is already verified
    | FieldDef << ast.EndField($0, nil, $Context >>
    ;

FieldDef : FieldName TypeName << ast.StartField($0, $1, $Context) >>;

(Obviously, the yacc implementation is single-threaded)

[awalterschulze]

I am not sure, but don't you have enough information in the token.Token, maybe I forgot what was in there?

[kfsone]

@awalterschulze just the literal and the Pos{line, col, offset}

I'm now using token context in my code to keep scope info that allows me to guide my parse while descending a relatively simple jsonish grammar and to allow go-style resolution of names between files without explicit imports using my "daycare" (... dependents) library

[awalterschulze]

So it seems token has everything except filename?

[kfsone]

Yeah, I just think it's perhaps overly specific for gocc as a general purpose parser generator.

I'd rather keep the token completely simple and use a Token factory that the user can override, so that there's no overhead for the average user and anyone who wants to get fancy just uses their own token type that complies with the gocc interface.

type MyToken struct {
    token.Token
    Filename string
    ASTStack []MyASTThings
}

type MyContext struct {
    fileSpecificFlags int
    errorCount int
    // ...
}

func NewMyToken(literal []byte, pos token.Pos, context interface{}) (*MyToken, error) {
  ...
}

func main() {
    p := parser.NewParser()
    l := lexer.NewLexerWithTokenFactory(getCode(), NewMyToken)
    p.ParseWithContext(l, &MyContext{})
    ...
}

[awalterschulze]

I think this is something that someone other than me would also need to look at @goccmack ?

[kfsone]

("previously on #111 ...") The two options I was positing:

• Add an opaque "Context" field to token.Pos, populated from the same in lexer.Lexer, and adding a $Context sdt variable, as I've done here https://github.com/kfsone/gocc/blob/dev/custom/internal/token/gen/golang/token.go or,
• Make it possible for the user to wrap extend Token themselves by using an overridable factory for Token construction.

[goccmack]

@kfsone Thanks for asking interesting questions. I get Page not found on https://github.com/kfsone/gocc/blob/dev/custom/internal/token/gen/golang/token.go

Could you make it available again?

[kfsone]

Ah, yes, sorry, I'd deleted the branch while rebasing.

https://github.com/kfsone/gocc/blob/feature/user-context/internal/token/gen/golang/token.go#L92

[kfsone]

Closing as I saw you merged the feature :)