search

gristlabs / asttokens

Annotate Python AST trees with source text and token information
Apache License 2.0
172 stars 34 forks source link

Explore a clearer lazy-parsing behaviour API #98

Closed PeterJCLaw closed 2 years ago

PeterJCLaw commented 2 years ago

This introduces a new class which has the lazy-parsing behaviour, separate from ASTTokens, thus leaving ASTTokens semantics unchanged.

ASTTokens still learns how to handle nodes unmarked, however it is left somewhat cautious in the use of them. The new class instead prefers them wherever it can.

This aims to provide an alternative to the API from #93 as the current API would make static analysis of ASTTokens much less useful when the init_tokens argument is used.

While it looks like there's a lot of change here, there isn't -- most of the diff is moving functions to the top of the file.

alexmojaki commented 2 years ago

Thanks, this looks promising. My PR is stable again, you can resolve the conflicts.

alexmojaki commented 2 years ago

@PeterJCLaw if you update this branch with the latest in #93 and consider this PR ok to merge, then I'll probably merge it. No need for a full review here, that can be done within #93.

alexmojaki commented 2 years ago

Thanks!