In Python3 Parser I am not able to get values of variable , function and class

[DivyaSrivastava45]

1. for Variables I am using atom_expr and atom but it is giving the values which are not considered as variable in python
2. for Functions which are not declared using def keyword , or e.g call functions no grammar for it ? please help me regarding this

[kaby76]

Which Python3 grammar? There are many, https://github.com/antlr/grammars-v4/tree/master/python

[DivyaSrivastava45]

I have used grammar folder --> grammars-v4/python/python3 generated grammar using there two files

1. python3lexer.g4
2. python3parser.g4 and used
3. python3lexerbase.java file inside java folder

[kaby76]

Do you have a Python3 input file that demonstrates the problem so as I can then generate the parse tree and discuss it?

[DivyaSrivastava45]

yeah we can discuss it on public repo https://github.com/rahulsain/Hackerrank_30daysOFcode.git
using 30daysOFcode_Python

[kaby76]

• The first thing to note is that some of the examples do not parse. That should be addressed, but not immediately.

  for i in ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/*.py; do echo "$i"; trparse "$i" > /dev/null; done

  ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/day 03.py line 1:2 no viable alternative at input '''

  ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/day 12.py line 40:30 no viable alternative at input ''

  ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/day 15.py line 28:23 no viable alternative at input ';'

  ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/day 22.py line 30:20 no viable alternative at input ''

  ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/day 24.py line 44:22 no viable alternative at input ';'

• I should also note that this grammar is (or at least should be) scraped directly from the CPython parser generator grammar that Python itself uses. Deviations in the grammar should be used sparingly.

• I use XPath to specify exactly and precisely what nodes in the parse tree I want to talk about. Compiler writers need to use a precise language to discuss problems. I implemented an XPath2 engine for Antlr4, and a cheatsheet for the syntax is here. It's implemented in a toolkit called Trash.

• Let's take "day 04.py".

  • The parse tree is here. tree.txt
  • The variables in this file, starting from the top, are: self (several methods), initialAge, t, i, age (not within the class, as that is a "field"), p, j.
  • As I think you noted, there are many `//atom_expr/atom/NAME' nodes that include nodes that are not considered variables, e.g., "print". For example, these should not include a "trailer" node that begins with a "(" because that denotes a function call.
  • So, trparse ../../../../Hackerrank_30daysOFcode/30daysOFcode_Python/day\ 04.py | trxgrep '//atom_expr/atom[not(following-sibling::trailer/OPEN_PAREN)]/NAME/text()' returns a list of nodes eliminating function calls: age, initialAge, self, initialAge, self, self, self, self, self, t, i, t, age, p, age, j. This list still contains more nodes that should not be there, e.g., "age" because that is a field of the class. You will need to further refine the XPath expression.

[DivyaSrivastava45]

Thank you for your help i understand the process but I never used xpath could you give me any source from where I can undertand xpath concept.

[kaby76]

XPath is a expression language used to navigate around a tree. See wikipedia.The W3Schools tutorial is okay. Also see my cheatsheet page, and Antlr's XPath implementation in the runtime. XPath has been around for decades, but it's a very specialized language. It came about because people were using XML, which is just another way to represent a tree, but needed a way to fetch data out of that tree.

The main point of my previous comment is that you need to consider the context of a particular node in the parse tree. You can't just pick off "atom" nodes and say that those are the identifiers.

If you are using an Antlr Visitor or Listener to find identifiers, then in the VisitAtom() method, you will need to navigate around the parse tree and verify the context of the "atom" node. You might be able to call XPath in the visitor method to do that, or just use the generated methods Antlr provides for your language.

So, for the expression '//atom_expr/atom[not(following-sibling::trailer/OPEN_PAREN)]/NAME/text()', this means "find an atom_expr node anywhere in the tree ('//'). At the 'atom_expr' node candidate, find a child labeled 'atom'. If there isn't one, then this 'atom_expr' doesn't contain an identifier. Then, get the sibling of the 'atom' node. Verify that it does not a child named 'trailer'. If it is, then this 'atom_expr' doesn't contain an identifier. ........"

[DivyaSrivastava45]

Thanks you . After Parsing I am getting " extraneous input ' ' expecting" at the end of files if there is an whitespace and because of this it is throwing syntax error is there anyway to handle this ?