Open hbutani opened 2 years ago
Hi,
the go runtime is going through a significant bug fix effort as we speak. Have you tried the below with another runtime ?
Eric
Le 22 août 2022 à 17:38, Harish Butani @.***> a écrit :
All ANTLR contributors provide implementation, maintenance and support for free. We therefore kindly ask that before submitting any issue to ANTLR, you please check off these boxes:
- [ Yes] I have reproduced my issue using the latest version of ANTLR
- [ No] I have asked at stackoverflow
- [ N/A] Responses from the above seem to indicate that my issue could be an ANTLR bug
- [Yes ] I have done a search of the existing issues to make sure I'm not sending in a duplicate
Please include the following information:
- target information: Go
- smallest possible grammar and code that reproduces the behavior. See below
- description of the expected behavior and actual behavior: See below Pointers to suspicious code regions are also very welcome. Relevant grammar rules:
statement : ... DROP deck_command CONNECTION identifier | CREATE deck_command CONNECTION identifier AS ACCT identifier ; deck_command : DECK {this.setDeckCommand()} ; In the generated grammar, I see the following.
For the Create deck choice(And most other choices): { p.SetState(529) p.Sundeck_command() } Whereas for the DROP deck choice I see: { p.SetState(525) p.Match(SqlBaseDECK) } So the behavior we observe is that in certain cases setDeckCommand() is not invoked.
Natural to think that there maybe an optimization that replaces a rule invocation with just a match. But there are some cases were optimization is not accounting for an action in the rule.
— Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/issues/3833, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZNQJH2AT3Y52YWDHL7S3TV2ONFXANCNFSM57IE4QTQ. You are receiving this because you are subscribed to this thread.
Thanks for the quick response. I will try with another runtime and get back(by end of day). But we need it to work in Go. Could you please point me to the effort on the significant bug fix effort.
@hbutani I can't reproduce whatever you are trying to convey.
Using Antlr4.10.1, I wrote a grammar that invokes an action.
Here is a grammar, "x.g4":
grammar x;
options {superClass=xParserBase;}
parse: statement* EOF;
statement :
DROP deck_command CONNECTION identifier
| CREATE deck_command CONNECTION identifier AS ACCT identifier
;
deck_command
: DECK {p.SetSet()}
;
identifier : IDENTIFIER;
ACCT: 'ACCT';
AS: 'AS';
CREATE: 'CREATE';
DECK: 'DECK';
DROP: 'DROP';
CONNECTION: 'CONNECTION';
IDENTIFIER : [a-zA-Z]+;
WS : [ \t\r\n]+ -> skip;
Here is the base class "xParserBase.go":
package parser
import (
"github.com/antlr/antlr4/runtime/Go/antlr"
)
type xParserBase struct {
*antlr.BaseParser
}
func (p *xParserBase) SetSet() bool {
return false
}
Here is the generated output:
// Code generated from parser/x.g4 by ANTLR 4.10.1. DO NOT EDIT.
package parser // x
import (
"fmt"
"strconv"
"sync"
"github.com/antlr/antlr4/runtime/Go/antlr"
)
// Suppress unused import errors
var _ = fmt.Printf
var _ = strconv.Itoa
var _ = sync.Once{}
type xParser struct {
xParserBase
}
var xParserStaticData struct {
once sync.Once
serializedATN []int32
literalNames []string
symbolicNames []string
ruleNames []string
predictionContextCache *antlr.PredictionContextCache
atn *antlr.ATN
decisionToDFA []*antlr.DFA
}
func xParserInit() {
staticData := &xParserStaticData
staticData.literalNames = []string{
"", "'ACCT'", "'AS'", "'CREATE'", "'DECK'", "'DROP'", "'CONNECTION'",
}
staticData.symbolicNames = []string{
"", "ACCT", "AS", "CREATE", "DECK", "DROP", "CONNECTION", "IDENTIFIER",
"WS",
}
staticData.ruleNames = []string{
"parse", "statement", "deck_command", "identifier",
}
staticData.predictionContextCache = antlr.NewPredictionContextCache()
staticData.serializedATN = []int32{
4, 1, 8, 37, 2, 0, 7, 0, 2, 1, 7, 1, 2, 2, 7, 2, 2, 3, 7, 3, 1, 0, 5, 0,
10, 8, 0, 10, 0, 12, 0, 13, 9, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1, 30, 8, 1, 1, 2,
1, 2, 1, 2, 1, 3, 1, 3, 1, 3, 0, 0, 4, 0, 2, 4, 6, 0, 0, 34, 0, 11, 1,
0, 0, 0, 2, 29, 1, 0, 0, 0, 4, 31, 1, 0, 0, 0, 6, 34, 1, 0, 0, 0, 8, 10,
3, 2, 1, 0, 9, 8, 1, 0, 0, 0, 10, 13, 1, 0, 0, 0, 11, 9, 1, 0, 0, 0, 11,
12, 1, 0, 0, 0, 12, 14, 1, 0, 0, 0, 13, 11, 1, 0, 0, 0, 14, 15, 5, 0, 0,
1, 15, 1, 1, 0, 0, 0, 16, 17, 5, 5, 0, 0, 17, 18, 3, 4, 2, 0, 18, 19, 5,
6, 0, 0, 19, 20, 3, 6, 3, 0, 20, 30, 1, 0, 0, 0, 21, 22, 5, 3, 0, 0, 22,
23, 3, 4, 2, 0, 23, 24, 5, 6, 0, 0, 24, 25, 3, 6, 3, 0, 25, 26, 5, 2, 0,
0, 26, 27, 5, 1, 0, 0, 27, 28, 3, 6, 3, 0, 28, 30, 1, 0, 0, 0, 29, 16,
1, 0, 0, 0, 29, 21, 1, 0, 0, 0, 30, 3, 1, 0, 0, 0, 31, 32, 5, 4, 0, 0,
32, 33, 6, 2, -1, 0, 33, 5, 1, 0, 0, 0, 34, 35, 5, 7, 0, 0, 35, 7, 1, 0,
0, 0, 2, 11, 29,
}
deserializer := antlr.NewATNDeserializer(nil)
staticData.atn = deserializer.Deserialize(staticData.serializedATN)
atn := staticData.atn
staticData.decisionToDFA = make([]*antlr.DFA, len(atn.DecisionToState))
decisionToDFA := staticData.decisionToDFA
for index, state := range atn.DecisionToState {
decisionToDFA[index] = antlr.NewDFA(state, index)
}
}
// xParserInit initializes any static state used to implement xParser. By default the
// static state used to implement the parser is lazily initialized during the first call to
// NewxParser(). You can call this function if you wish to initialize the static state ahead
// of time.
func XParserInit() {
staticData := &xParserStaticData
staticData.once.Do(xParserInit)
}
// NewxParser produces a new parser instance for the optional input antlr.TokenStream.
func NewxParser(input antlr.TokenStream) *xParser {
XParserInit()
this := new(xParser)
this.BaseParser = antlr.NewBaseParser(input)
staticData := &xParserStaticData
this.Interpreter = antlr.NewParserATNSimulator(this, staticData.atn, staticData.decisionToDFA, staticData.predictionContextCache)
this.RuleNames = staticData.ruleNames
this.LiteralNames = staticData.literalNames
this.SymbolicNames = staticData.symbolicNames
this.GrammarFileName = "x.g4"
return this
}
// xParser tokens.
const (
xParserEOF = antlr.TokenEOF
xParserACCT = 1
xParserAS = 2
xParserCREATE = 3
xParserDECK = 4
xParserDROP = 5
xParserCONNECTION = 6
xParserIDENTIFIER = 7
xParserWS = 8
)
// xParser rules.
const (
xParserRULE_parse = 0
xParserRULE_statement = 1
xParserRULE_deck_command = 2
xParserRULE_identifier = 3
)
// IParseContext is an interface to support dynamic dispatch.
type IParseContext interface {
antlr.ParserRuleContext
// GetParser returns the parser.
GetParser() antlr.Parser
// IsParseContext differentiates from other interfaces.
IsParseContext()
}
type ParseContext struct {
*antlr.BaseParserRuleContext
parser antlr.Parser
}
func NewEmptyParseContext() *ParseContext {
var p = new(ParseContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(nil, -1)
p.RuleIndex = xParserRULE_parse
return p
}
func (*ParseContext) IsParseContext() {}
func NewParseContext(parser antlr.Parser, parent antlr.ParserRuleContext, invokingState int) *ParseContext {
var p = new(ParseContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(parent, invokingState)
p.parser = parser
p.RuleIndex = xParserRULE_parse
return p
}
func (s *ParseContext) GetParser() antlr.Parser { return s.parser }
func (s *ParseContext) EOF() antlr.TerminalNode {
return s.GetToken(xParserEOF, 0)
}
func (s *ParseContext) AllStatement() []IStatementContext {
children := s.GetChildren()
len := 0
for _, ctx := range children {
if _, ok := ctx.(IStatementContext); ok {
len++
}
}
tst := make([]IStatementContext, len)
i := 0
for _, ctx := range children {
if t, ok := ctx.(IStatementContext); ok {
tst[i] = t.(IStatementContext)
i++
}
}
return tst
}
func (s *ParseContext) Statement(i int) IStatementContext {
var t antlr.RuleContext
j := 0
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IStatementContext); ok {
if j == i {
t = ctx.(antlr.RuleContext)
break
}
j++
}
}
if t == nil {
return nil
}
return t.(IStatementContext)
}
func (s *ParseContext) GetRuleContext() antlr.RuleContext {
return s
}
func (s *ParseContext) ToStringTree(ruleNames []string, recog antlr.Recognizer) string {
return antlr.TreesStringTree(s, ruleNames, recog)
}
func (s *ParseContext) EnterRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.EnterParse(s)
}
}
func (s *ParseContext) ExitRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.ExitParse(s)
}
}
func (p *xParser) Parse() (localctx IParseContext) {
this := p
_ = this
localctx = NewParseContext(p, p.GetParserRuleContext(), p.GetState())
p.EnterRule(localctx, 0, xParserRULE_parse)
var _la int
defer func() {
p.ExitRule()
}()
defer func() {
if err := recover(); err != nil {
if v, ok := err.(antlr.RecognitionException); ok {
localctx.SetException(v)
p.GetErrorHandler().ReportError(p, v)
p.GetErrorHandler().Recover(p, v)
} else {
panic(err)
}
}
}()
p.EnterOuterAlt(localctx, 1)
p.SetState(11)
p.GetErrorHandler().Sync(p)
_la = p.GetTokenStream().LA(1)
for _la == xParserCREATE || _la == xParserDROP {
{
p.SetState(8)
p.Statement()
}
p.SetState(13)
p.GetErrorHandler().Sync(p)
_la = p.GetTokenStream().LA(1)
}
{
p.SetState(14)
p.Match(xParserEOF)
}
return localctx
}
// IStatementContext is an interface to support dynamic dispatch.
type IStatementContext interface {
antlr.ParserRuleContext
// GetParser returns the parser.
GetParser() antlr.Parser
// IsStatementContext differentiates from other interfaces.
IsStatementContext()
}
type StatementContext struct {
*antlr.BaseParserRuleContext
parser antlr.Parser
}
func NewEmptyStatementContext() *StatementContext {
var p = new(StatementContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(nil, -1)
p.RuleIndex = xParserRULE_statement
return p
}
func (*StatementContext) IsStatementContext() {}
func NewStatementContext(parser antlr.Parser, parent antlr.ParserRuleContext, invokingState int) *StatementContext {
var p = new(StatementContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(parent, invokingState)
p.parser = parser
p.RuleIndex = xParserRULE_statement
return p
}
func (s *StatementContext) GetParser() antlr.Parser { return s.parser }
func (s *StatementContext) DROP() antlr.TerminalNode {
return s.GetToken(xParserDROP, 0)
}
func (s *StatementContext) Deck_command() IDeck_commandContext {
var t antlr.RuleContext
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IDeck_commandContext); ok {
t = ctx.(antlr.RuleContext)
break
}
}
if t == nil {
return nil
}
return t.(IDeck_commandContext)
}
func (s *StatementContext) CONNECTION() antlr.TerminalNode {
return s.GetToken(xParserCONNECTION, 0)
}
func (s *StatementContext) AllIdentifier() []IIdentifierContext {
children := s.GetChildren()
len := 0
for _, ctx := range children {
if _, ok := ctx.(IIdentifierContext); ok {
len++
}
}
tst := make([]IIdentifierContext, len)
i := 0
for _, ctx := range children {
if t, ok := ctx.(IIdentifierContext); ok {
tst[i] = t.(IIdentifierContext)
i++
}
}
return tst
}
func (s *StatementContext) Identifier(i int) IIdentifierContext {
var t antlr.RuleContext
j := 0
for _, ctx := range s.GetChildren() {
if _, ok := ctx.(IIdentifierContext); ok {
if j == i {
t = ctx.(antlr.RuleContext)
break
}
j++
}
}
if t == nil {
return nil
}
return t.(IIdentifierContext)
}
func (s *StatementContext) CREATE() antlr.TerminalNode {
return s.GetToken(xParserCREATE, 0)
}
func (s *StatementContext) AS() antlr.TerminalNode {
return s.GetToken(xParserAS, 0)
}
func (s *StatementContext) ACCT() antlr.TerminalNode {
return s.GetToken(xParserACCT, 0)
}
func (s *StatementContext) GetRuleContext() antlr.RuleContext {
return s
}
func (s *StatementContext) ToStringTree(ruleNames []string, recog antlr.Recognizer) string {
return antlr.TreesStringTree(s, ruleNames, recog)
}
func (s *StatementContext) EnterRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.EnterStatement(s)
}
}
func (s *StatementContext) ExitRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.ExitStatement(s)
}
}
func (p *xParser) Statement() (localctx IStatementContext) {
this := p
_ = this
localctx = NewStatementContext(p, p.GetParserRuleContext(), p.GetState())
p.EnterRule(localctx, 2, xParserRULE_statement)
defer func() {
p.ExitRule()
}()
defer func() {
if err := recover(); err != nil {
if v, ok := err.(antlr.RecognitionException); ok {
localctx.SetException(v)
p.GetErrorHandler().ReportError(p, v)
p.GetErrorHandler().Recover(p, v)
} else {
panic(err)
}
}
}()
p.SetState(29)
p.GetErrorHandler().Sync(p)
switch p.GetTokenStream().LA(1) {
case xParserDROP:
p.EnterOuterAlt(localctx, 1)
{
p.SetState(16)
p.Match(xParserDROP)
}
{
p.SetState(17)
p.Deck_command()
}
{
p.SetState(18)
p.Match(xParserCONNECTION)
}
{
p.SetState(19)
p.Identifier()
}
case xParserCREATE:
p.EnterOuterAlt(localctx, 2)
{
p.SetState(21)
p.Match(xParserCREATE)
}
{
p.SetState(22)
p.Deck_command()
}
{
p.SetState(23)
p.Match(xParserCONNECTION)
}
{
p.SetState(24)
p.Identifier()
}
{
p.SetState(25)
p.Match(xParserAS)
}
{
p.SetState(26)
p.Match(xParserACCT)
}
{
p.SetState(27)
p.Identifier()
}
default:
panic(antlr.NewNoViableAltException(p, nil, nil, nil, nil, nil))
}
return localctx
}
// IDeck_commandContext is an interface to support dynamic dispatch.
type IDeck_commandContext interface {
antlr.ParserRuleContext
// GetParser returns the parser.
GetParser() antlr.Parser
// IsDeck_commandContext differentiates from other interfaces.
IsDeck_commandContext()
}
type Deck_commandContext struct {
*antlr.BaseParserRuleContext
parser antlr.Parser
}
func NewEmptyDeck_commandContext() *Deck_commandContext {
var p = new(Deck_commandContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(nil, -1)
p.RuleIndex = xParserRULE_deck_command
return p
}
func (*Deck_commandContext) IsDeck_commandContext() {}
func NewDeck_commandContext(parser antlr.Parser, parent antlr.ParserRuleContext, invokingState int) *Deck_commandContext {
var p = new(Deck_commandContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(parent, invokingState)
p.parser = parser
p.RuleIndex = xParserRULE_deck_command
return p
}
func (s *Deck_commandContext) GetParser() antlr.Parser { return s.parser }
func (s *Deck_commandContext) DECK() antlr.TerminalNode {
return s.GetToken(xParserDECK, 0)
}
func (s *Deck_commandContext) GetRuleContext() antlr.RuleContext {
return s
}
func (s *Deck_commandContext) ToStringTree(ruleNames []string, recog antlr.Recognizer) string {
return antlr.TreesStringTree(s, ruleNames, recog)
}
func (s *Deck_commandContext) EnterRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.EnterDeck_command(s)
}
}
func (s *Deck_commandContext) ExitRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.ExitDeck_command(s)
}
}
func (p *xParser) Deck_command() (localctx IDeck_commandContext) {
this := p
_ = this
localctx = NewDeck_commandContext(p, p.GetParserRuleContext(), p.GetState())
p.EnterRule(localctx, 4, xParserRULE_deck_command)
defer func() {
p.ExitRule()
}()
defer func() {
if err := recover(); err != nil {
if v, ok := err.(antlr.RecognitionException); ok {
localctx.SetException(v)
p.GetErrorHandler().ReportError(p, v)
p.GetErrorHandler().Recover(p, v)
} else {
panic(err)
}
}
}()
p.EnterOuterAlt(localctx, 1)
{
p.SetState(31)
p.Match(xParserDECK)
}
p.SetSet()
return localctx
}
// IIdentifierContext is an interface to support dynamic dispatch.
type IIdentifierContext interface {
antlr.ParserRuleContext
// GetParser returns the parser.
GetParser() antlr.Parser
// IsIdentifierContext differentiates from other interfaces.
IsIdentifierContext()
}
type IdentifierContext struct {
*antlr.BaseParserRuleContext
parser antlr.Parser
}
func NewEmptyIdentifierContext() *IdentifierContext {
var p = new(IdentifierContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(nil, -1)
p.RuleIndex = xParserRULE_identifier
return p
}
func (*IdentifierContext) IsIdentifierContext() {}
func NewIdentifierContext(parser antlr.Parser, parent antlr.ParserRuleContext, invokingState int) *IdentifierContext {
var p = new(IdentifierContext)
p.BaseParserRuleContext = antlr.NewBaseParserRuleContext(parent, invokingState)
p.parser = parser
p.RuleIndex = xParserRULE_identifier
return p
}
func (s *IdentifierContext) GetParser() antlr.Parser { return s.parser }
func (s *IdentifierContext) IDENTIFIER() antlr.TerminalNode {
return s.GetToken(xParserIDENTIFIER, 0)
}
func (s *IdentifierContext) GetRuleContext() antlr.RuleContext {
return s
}
func (s *IdentifierContext) ToStringTree(ruleNames []string, recog antlr.Recognizer) string {
return antlr.TreesStringTree(s, ruleNames, recog)
}
func (s *IdentifierContext) EnterRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.EnterIdentifier(s)
}
}
func (s *IdentifierContext) ExitRule(listener antlr.ParseTreeListener) {
if listenerT, ok := listener.(xListener); ok {
listenerT.ExitIdentifier(s)
}
}
func (p *xParser) Identifier() (localctx IIdentifierContext) {
this := p
_ = this
localctx = NewIdentifierContext(p, p.GetParserRuleContext(), p.GetState())
p.EnterRule(localctx, 6, xParserRULE_identifier)
defer func() {
p.ExitRule()
}()
defer func() {
if err := recover(); err != nil {
if v, ok := err.(antlr.RecognitionException); ok {
localctx.SetException(v)
p.GetErrorHandler().ReportError(p, v)
p.GetErrorHandler().Recover(p, v)
} else {
panic(err)
}
}
}()
p.EnterOuterAlt(localctx, 1)
{
p.SetState(34)
p.Match(xParserIDENTIFIER)
}
return localctx
}
This code compiles and runs perfectly fine, Antlr 4.10.1, Go 1.19.
In the generated parser.go file, there is only one call to "SetSet()", and that is within func "Deck_command()".
Please explain to me what I am seeing that is wrong. What I should be seeing? What "optimization" are you referring to?
Hey @kaby76 thanks for doing a detailed example. The issue is not easily reproducible. It seems related to the specific grammar rules.
In your example:
in both cases of func (p *xParser) Statement() (localctx IStatementContext) {..
the p.Deck_command()
is called.
So you have:
{
p.SetState(17)
p.Deck_command()
}
...
{
p.SetState(22)
p.Deck_command()
}
My grammar has many choices in the statement
rule, say 100. Out of that around 10 choices refer to the deck_command
command. Among the 10, the code generated for a majority invokes p.Deck_command()
.
But I see a couple where instead p.Match(SqlBaseDECK)
gets called.
Its hard for me to share the entire grammar. I was wondering if you guys(the developers) can spot a case where the code generated is generating the p.Match(SqlBaseDECK)
inspite of there being action code in the rule.
There's no way we can figure this out without a grammar. My guess is that you have more than one applied occurrence of the DECK
symbol in your grammar.
Relevant grammar rules:
In the generated grammar, I see the following.
Create deck
choice(And most other choices):DROP deck
choice I see:So the behavior we observe is that in certain cases
setDeckCommand()
is not invoked.Natural to think that there maybe an optimization that replaces a rule invocation with just a match. But there are some cases were optimization is not accounting for an action in the rule.