fix(parser): exclude empty tokens and cstRoot bug

[rodrigo-brito]

• I converted the parser to unix format, to support Windows + Linux + Mac
• root.addNode(cstNode); should be called to all CST Nodes.
• I include some unit tests to validate the parser.
• I also excluded all empty tokens, such as empty spaces, end lines (it affects the precision and recall)

One test failed:

For the example in the test-data/parser/python/example.py, the parent of the function sum and multi must be the class test and not example.py, right?

[rodrigo-brito]

Now, the plugin detects MOVE and RENAME

[Mosallamy]

Hi @rodrigo-brito , thanks a lot for your fixes

As for your question, yes test should be the parent of both (sum and multi) and not example.py

But I have a question, in this part we added this condition because the root node should contain only the file.

In the example bellow, the file contains a class and the class contains a function, the root should have one node inside it which is the file and the class will be one of the nodes inside the file and the function will be one of the nodes inside the class

[rodrigo-brito]

If you look in unit tests for Java Plugin, you will see that CST Root should contain all nodes, it is a kind of index of nodes: https://github.com/aserg-ufmg/RefDiff/blob/889b0bfbf2c18726d44f077371966606232cca0b/refdiff-java/src/test/java/refdiff/parsers/java/TestJavaParser.java#L33

The hierarchy of nodes is defined by the parent field, which specifies the parent if it is a child node, and for the root node, we do not have a parent.

The second option. We can define all children's node. For example

• example.py
  • Test
    • sum
    • multi

[Mosallamy]

The CST below is generated for the code example in your repo: https://github.com/rodrigo-brito/refactoring-python-example

and it works as you described, the function printIsEven is a children node for the main.py

[rodrigo-brito]

Yes, it looks fine. I will revert this change.

[rodrigo-brito]

Running the tests. Apparently, it includes only the last node, the node three (sum function is missing):

[Mosallamy]

Oh! I'will check it out and get back to you, maybe the problem is in the parser

[rodrigo-brito]

I will open a pull request with the revert and some fix in unit tests and try to find the bug too.

[QassemNa]

Hi everyone, I have made a pull request to solve the issue for only the last node appering