Previously, code_tokenize employed custom configurable tokenizers to parse code. However, the implementation is limited by design and required work around for language dependent changes.
The new API introduces the visitor pattern for tokenization. Dependent on the language that should parsed, we can employ custom visitors to parse the AST.
In addition, AST parsing is now outsourced to a custom library (specialized for the AST handling process)
CHANGE
Previously, code_tokenize employed custom configurable tokenizers to parse code. However, the implementation is limited by design and required work around for language dependent changes.
The new API introduces the visitor pattern for tokenization. Dependent on the language that should parsed, we can employ custom visitors to parse the AST.
In addition, AST parsing is now outsourced to a custom library (specialized for the AST handling process)