search

pboyer / antlr4

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
http://antlr.org
Other
26 stars 1 forks source link

antlr4gen4go tool name change proposal, install, setup and usage #36

Closed omac777 closed 8 years ago

omac777 commented 8 years ago

Your fork of antlr to generate golang code is awesome. Thank you.

  1. 1)

I propose you rename it to distinguish it from the original antlr tool and and clarify its distinct intended purpose.

  1. 2)

On Debian, you'll need to install a few things apt-get install openjdk-8-jdk apt-get install maven go get github.com/pboyer/antlr4 cd /usr/local/lib curl -O http://www.stringtemplate.org/download/ST-4.0.8.jar export CLASSPATH=".:/usr/local/lib/ST-4.0.8.jar:$CLASSPATH"

  1. 3)

If you aren't accustomed to maven, it's important to specify where to invoke the command. In this case, it's important to invoke the mvn install command from the top directory of pboyer's antlr4: cd $GOPATH/src/github.com/pboyer/antlr4/ mvn install NOTE: the above built, installed and tested antlr4 gen4go. It also left the residual tests in /tmp/. I would recommend you go there to peruse the Test.go files to inspire you for main drivers for golang that utilize the parsers.

  1. 4)

Initialize some other environment variables pointing to the newly built antlr4 gen4go: export CLASSPATH=".:$GOPATH/src/github.com/pboyer/antlr4/tool/target/antlr4-4.5.2-SNAPSHOT.jar:$CLASSPATH" alias antlr='java -jar $GOPATH/src/github.com/pboyer/antlr4/tool/target/antlr4-4.5.2-SNAPSHOT.jar'

  1. 5)

FYI WITHIN THIS DIRECTORY: $GOPATH/src/github.com/pboyer/antlr4/runtime-testsuite/test/org/antlr/v4/test/runtime/go/ There are tests written in java that generate "Test.go" containing the sample maindriver. They create a bunch a directories within /tmp i.e.

Usage:

  1. 6)

Go to one of the test full context parsing directories within the /tmp directory: /tmp/TestFullContextParsing-1467499156161# ls -lt input-----IS A FILE HOLDING TEST INPUT parser------IS A DIRECTORY HOLDING THE T.g4 and generated golang files Test.go-----IS A FILE HOLDING THE MAINDRIVER CALLING PARSER HELD IN ./parser/

  1. 6.1)

Here is an example main driver as generated by pboyer/antlr4/runtime-testsuite/test/org/antlr/v4/test/runtime/go/: Test.go: package main import ( "github.com/pboyer/antlr4/runtime/Go/antlr" "./parser" "os" )

type TreeShapeListener struct { *parser.BaseTListener }

func NewTreeShapeListener() *TreeShapeListener { return new(TreeShapeListener) }

func (this *TreeShapeListener) EnterEveryRule(ctx antlr.ParserRuleContext) { for i := 0; i<ctx.GetChildCount(); i++ { child := ctx.GetChild(i) parentR,ok := child.GetParent().(antlr.RuleNode) if !ok || parentR.GetBaseRuleContext() != ctx.GetBaseRuleContext() { panic("Invalid parse tree shape detected.") } } }

func main() { input := antlr.NewFileStream(os.Args[1]) lexer := parser.NewTLexer(input) stream := antlr.NewCommonTokenStream(lexer,0) p := parser.NewTParser(stream) p.AddErrorListener(antlr.NewDiagnosticErrorListener(true)) p.BuildParseTrees = true tree := p.Prog() antlr.ParseTreeWalkerDefault.Walk(NewTreeShapeListener(), tree) }

  1. 6.2)

Here are the contents of the test input file: input: a(i)<-x

  1. 6.3)

Here are the contents of the ./parser/T.g4 grammar that was generated into golang: T.g4: more T.g4 grammar T; prog @init {p.Interpreter.SetPredictionMode(antlr.PredictionModeLLExactAmbigDetection);} : expr_or_assign*; expr_or_assign : expr '++' {fmt.Println("fail.")} | expr {fmt.Println("pass: "+$expr.text)} ; expr: expr_primary ('<-' ID)?; expr_primary : '(' ID ')' | ID '(' ID ')' | ID ; ID : [a-z]+ ;

  1. 6.4)

Generate the golang go for the grammar described in T.g4: cd /tmp/TestFullContextParsing-1467499156161/ antlr T.g4 -Dlanguage=Go

  1. 6.5)

Then build the Test.go maindriver's binary: go build

  1. 6.6)

Then test the parser: TestFullContextParsing-1467499156161 input

That's it.

I do hope it clarifies how to use this AWESOME GO LANGUAGE TOOL!

willfaught commented 8 years ago

Hey David, glad to see another Go/ANTLR enthusiast!

I'm having trouble understanding some things. Are you suggesting renaming the repo, the command name, or both? Why is it important to distinguish it from the original ANTLR? Isn't its intended purpose the same (generating code)? Are you suggesting we add Debian-specific instructions to the README?

Maybe this will clear up a lot of these points: the intent of this fork is to add Go as one more of many code generation language targets. ANTLR can already generate Python2/3, C#, JavaScript, and Java. So Go will just be one more option. The changes in this repo will be pulled back into the main ANTLR repo when they're ready.

The way the tests work was determined in the main fork, so we just add to and fix them as-is.

If you feel adding a note to the README about running maven from the root would be helpful, I recommend opening an issue for that in antlr/antlr4.

Hope that answers some things. Please reply if I didn't address something!

On Jul 2, 2016, at 5:55 PM, David Marceau notifications@github.com wrote:

Your fork of antlr to generate golang code is awesome. Thank you.

1) I propose you rename it to distinguish it from the original antlr tool and and clarify its distinct intended purpose.

2) On Debian, you'll need to install a few things apt-get install openjdk-8-jdk apt-get install maven go get github.com/pboyer/antlr4 cd /usr/local/lib curl -O http://www.stringtemplate.org/download/ST-4.0.8.jar export CLASSPATH=".:/usr/local/lib/ST-4.0.8.jar:$CLASSPATH"

3) If you aren't accustomed to maven, it's important to specify where to invoke the command. In this case, it's important to invoke the mvn install command from the top directory of pboyer's antlr4: cd $GOPATH/src/github.com/pboyer/antlr4/ mvn install NOTE: the above built, installed and tested antlr4 gen4go. It also left the residual tests in /tmp/. I would recommend you go there to peruse the Test.go files to inspire you for main drivers for golang that utilize the parsers.

4) Initialize some other environment variables pointing to the newly built antlr4 gen4go: export CLASSPATH=".:$GOPATH/src/github.com/pboyer/antlr4/tool/target/antlr4-4.5.2-SNAPSHOT.jar:$CLASSPATH" alias antlr='java -jar $GOPATH/src/github.com/pboyer/antlr4/tool/target/antlr4-4.5.2-SNAPSHOT.jar'

5) FYI WITHIN THIS DIRECTORY: $GOPATH/src/github.com/pboyer/antlr4/runtime-testsuite/test/org/antlr/v4/test/runtime/go/ There are tests written in java that generate "Test.go" containing the sample maindriver. They create a bunch a directories within /tmp i.e.

Usage:

6) Go to one of the test full context parsing directories within the /tmp directory: /tmp/TestFullContextParsing-1467499156161# ls -lt input-----IS A FILE HOLDING TEST INPUT parser------IS A DIRECTORY HOLDING THE T.g4 and generated golang files Test.go-----IS A FILE HOLDING THE MAINDRIVER CALLING PARSER HELD IN ./parser/

6.1) Here is an example main driver as generated by pboyer/antlr4/runtime-testsuite/test/org/antlr/v4/test/runtime/go/: Test.go: package main import ( "github.com/pboyer/antlr4/runtime/Go/antlr" "./parser" "os" )

type TreeShapeListener struct { *parser.BaseTListener }

func NewTreeShapeListener() *TreeShapeListener { return new(TreeShapeListener) }

func (this *TreeShapeListener) EnterEveryRule(ctx antlr.ParserRuleContext) { for i := 0; i<ctx.GetChildCount(); i++ { child := ctx.GetChild(i) parentR,ok := child.GetParent().(antlr.RuleNode) if !ok || parentR.GetBaseRuleContext() != ctx.GetBaseRuleContext() { panic("Invalid parse tree shape detected.") } } }

func main() { input := antlr.NewFileStream(os.Args[1]) lexer := parser.NewTLexer(input) stream := antlr.NewCommonTokenStream(lexer,0) p := parser.NewTParser(stream) p.AddErrorListener(antlr.NewDiagnosticErrorListener(true)) p.BuildParseTrees = true tree := p.Prog() antlr.ParseTreeWalkerDefault.Walk(NewTreeShapeListener(), tree) }

6.2) Here are the contents of the test input file: input: a(i)<-x

6.3) Here are the contents of the ./parser/T.g4 grammar that was generated into golang: T.g4: more T.g4 grammar T; prog @init {p.Interpreter.SetPredictionMode(antlr.PredictionModeLLExactAmbigDetection);} : expr_or_assign*; expr_or_assign : expr '++' {fmt.Println("fail.")} | expr {fmt.Println("pass: "+$expr.text)} ; expr: expr_primary ('<-' ID)?; expr_primary : '(' ID ')' | ID '(' ID ')' | ID ; ID : [a-z]+ ;

6.4) Generate the golang go for the grammar described in T.g4: cd /tmp/TestFullContextParsing-1467499156161/ antlr T.g4 -Dlanguage=Go

6.5) Then build the Test.go maindriver's binary: go build

6.6) Then test the parser: TestFullContextParsing-1467499156161 input

That's it.

I do hope it clarifies how to use this AWESOME GO LANGUAGE TOOL!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

omac777 commented 8 years ago

On 07/04/2016 03:59 PM, Will Faught wrote:

Hey David, glad to see another Go/ANTLR enthusiast!

I'm having trouble understanding some things. Are you suggesting renaming the repo, the command name, or both? Why is it important to distinguish it from the original ANTLR? Isn't its intended purpose the same (generating code)? Are you suggesting we add Debian-specific instructions to the README?

Maybe this will clear up a lot of these points: the intent of this fork is to add Go as one more of many code generation language targets. ANTLR can already generate Python2/3, C#, JavaScript, and Java. So Go will just be one more option. The changes in this repo will be pulled back into the main ANTLR repo when they're ready.

I interpreted the fork was created because the main developers had no interest in adding golang to antlr and they stated that in an email on a mailing list somewhere. Related to this matter Mr. Harwell wrote goworks, but it only supports netbeans7.3beta2. I tried netbeans7.3, 8.0 and 8.1 with the goworks plugin with openjdk debian sid(javac 1.8.0_91) but all of them complained about missing editor settings storage and related classes.

The way the tests work was determined in the main fork, so we just add to and fix them as-is. Ok.

If you feel adding a note to the README about running maven from the root would be helpful, I recommend opening an issue for that in antlr/antlr4. Ok.

One other thing. I believe goworks should be directly mentioned within antlr4 when they do merge golang into it.

The java stuff generates main drivers for go when doing the runtime-tests, but there are no documented example main drivers for intended audiences to quickly get up to speed with antlr4/golang. The only thing I can mention quickly is to check out the mvn test generated sources and look at the Test.go files and their associated ./parser/*.go files. Also to especially mention the token.go file to find what the users may fetch from the ctx and associated tokens. i.e. line/column/start/end/etc... With all this information antlr/golang wannabe's will get up to speed more quickly.

Hope that answers some things. Please reply if I didn't address something!

On Jul 2, 2016, at 5:55 PM, David Marceau notifications@github.com wrote:

Your fork of antlr to generate golang code is awesome. Thank you.

1) I propose you rename it to distinguish it from the original antlr tool and and clarify its distinct intended purpose.

2) On Debian, you'll need to install a few things apt-get install openjdk-8-jdk apt-get install maven go get github.com/pboyer/antlr4 cd /usr/local/lib curl -O http://www.stringtemplate.org/download/ST-4.0.8.jar export CLASSPATH=".:/usr/local/lib/ST-4.0.8.jar:$CLASSPATH"

3) If you aren't accustomed to maven, it's important to specify where to invoke the command. In this case, it's important to invoke the mvn install command from the top directory of pboyer's antlr4: cd $GOPATH/src/github.com/pboyer/antlr4/ mvn install NOTE: the above built, installed and tested antlr4 gen4go. It also left the residual tests in /tmp/. I would recommend you go there to peruse the Test.go files to inspire you for main drivers for golang that utilize the parsers.

4) Initialize some other environment variables pointing to the newly built antlr4 gen4go: export CLASSPATH=".:$GOPATH/src/github.com/pboyer/antlr4/tool/target/antlr4-4.5.2-SNAPSHOT.jar:$CLASSPATH" alias antlr='java -jar $GOPATH/src/github.com/pboyer/antlr4/tool/target/antlr4-4.5.2-SNAPSHOT.jar'

5) FYI WITHIN THIS DIRECTORY: $GOPATH/src/github.com/pboyer/antlr4/runtime-testsuite/test/org/antlr/v4/test/runtime/go/ There are tests written in java that generate "Test.go" containing the sample maindriver. They create a bunch a directories within /tmp i.e.

Usage:

6) Go to one of the test full context parsing directories within the /tmp directory: /tmp/TestFullContextParsing-1467499156161# ls -lt input-----IS A FILE HOLDING TEST INPUT parser------IS A DIRECTORY HOLDING THE T.g4 and generated golang files Test.go-----IS A FILE HOLDING THE MAINDRIVER CALLING PARSER HELD IN ./parser/

6.1) Here is an example main driver as generated by pboyer/antlr4/runtime-testsuite/test/org/antlr/v4/test/runtime/go/: Test.go: package main import ( "github.com/pboyer/antlr4/runtime/Go/antlr" "./parser" "os" )

type TreeShapeListener struct { *parser.BaseTListener }

func NewTreeShapeListener() *TreeShapeListener { return new(TreeShapeListener) }

func (this *TreeShapeListener) EnterEveryRule(ctx antlr.ParserRuleContext) { for i := 0; i<ctx.GetChildCount(); i++ { child := ctx.GetChild(i) parentR,ok := child.GetParent().(antlr.RuleNode) if !ok || parentR.GetBaseRuleContext() != ctx.GetBaseRuleContext() { panic("Invalid parse tree shape detected.") } } }

func main() { input := antlr.NewFileStream(os.Args[1]) lexer := parser.NewTLexer(input) stream := antlr.NewCommonTokenStream(lexer,0) p := parser.NewTParser(stream) p.AddErrorListener(antlr.NewDiagnosticErrorListener(true)) p.BuildParseTrees = true tree := p.Prog() antlr.ParseTreeWalkerDefault.Walk(NewTreeShapeListener(), tree) }

6.2) Here are the contents of the test input file: input: a(i)<-x

6.3) Here are the contents of the ./parser/T.g4 grammar that was generated into golang: T.g4: more T.g4 grammar T; prog @init {p.Interpreter.SetPredictionMode(antlr.PredictionModeLLExactAmbigDetection);} : expr_or_assign*; expr_or_assign : expr '++' {fmt.Println("fail.")} | expr {fmt.Println("pass: "+$expr.text)} ; expr: expr_primary ('<-' ID)?; expr_primary : '(' ID ')' | ID '(' ID ')' | ID ; ID : [a-z]+ ;

6.4) Generate the golang go for the grammar described in T.g4: cd /tmp/TestFullContextParsing-1467499156161/ antlr T.g4 -Dlanguage=Go

6.5) Then build the Test.go maindriver's binary: go build

6.6) Then test the parser: TestFullContextParsing-1467499156161 input

That's it.

I do hope it clarifies how to use this AWESOME GO LANGUAGE TOOL!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/pboyer/antlr4/issues/36#issuecomment-230347986

willfaught commented 8 years ago

I interpreted the fork was created because the main developers had no interest in adding golang to antlr and they stated that in an email on a mailing list somewhere

@sharwell wrote in https://github.com/antlr/antlr4/issues/659 on why he abandoned his effort to add a Go target:

I had some doubts about the future of Go. The port would be an enormous amount of work that I couldn't be sure would have long-term benefits, especially compared to some of the other targets which I can be involved with.

So it was just the way things worked out, not a strong opinion against adding Go support.

I believe goworks should be directly mentioned within antlr4 when they do merge golang into it.

It seems simpler to me to keep the repo application-agnostic and leave it to those projects that use ANTLR to document its use. I don't believe this kind of tool callout is done for other language targets, so antlr/antlr4 might have a problem with this being inconsistent. This is probably best tackled by filing an issue in antlr/antlr4 to see what they recommend.

The java stuff generates main drivers for go when doing the runtime-tests, but there are no documented example main drivers for intended audiences to quickly get up to speed with antlr4/golang. The only thing I can mention quickly is to check out the mvn test generated sources and look at the Test.go files and their associated ./parser/*.go files. Also to especially mention the token.go file to find what the users may fetch from the ctx and associated tokens. i.e. line/column/start/end/etc... With all this information antlr/golang wannabe's will get up to speed more quickly.

I completely understand the need to know how to get stuff working quickly. Not much of the runtime/Go/antlr package is documented right now, so if you have the interest and motivation, adding documentation for all the exported items there would be immensely helpful. The package doc would be a great place to put an example main() driver.

(Edited, again)

pboyer commented 8 years ago

Thanks @omac777 for the feedback. I think the comments above do a good job of representing how to debug the test failures. I'm going to close this. I think @willfaught captured the relevant points.