Open 5c4lar opened 2 years ago
Use a custom visitor like this:
class CLeafVisitor(ctok.tokenizer.LeafVisitor):
def visit_string_literal(self, node):
self.node_handler(node)
return False
seems to fix the problem
Thank you for this hint!
I will add more custom visitors for the supported languages in the next release. Until then, you can use custom visitors to parse your code. For example, you could use your C visitor as follows:
ctok.tokenize(sample, lang = "cpp", visitors=[CLeafVisitor])
For the following code
Output:
But parsing string literals works fine for Java and Python code. How should I fix this problem?