search

apryor6 / flaskerize

Python CLI build/dev tool for templated code generation and project modification. Think Angular schematics for Python.
BSD 3-Clause "New" or "Revised" License
121 stars 13 forks source link

Use Python AST for rewrites #8

Open apryor6 opened 5 years ago

apryor6 commented 5 years ago

For tasks involving modification of existing files, we should move to Python's abstract syntax tree module (AST).

apryor6 commented 5 years ago

Some context:

The existing functionality for code manipulation is directly inspecting and manipulating the string contents of the source code, but this has a number of problems and is very brittle.

It's perhaps easy to see why manually manipulating source code strings is very messy. A simple goal of wanting to find the call to Flask and check for a parameter, creating one if it does not exist requires considering multi-line definitions, parameters provided by keywords, comments at any point, whether or not the invocation is inside a function or uses another function or imports another function from a different file, etc. It is extremely difficult to catch all edge cases, and I am not intending to pursue this methodology further as it is completely impractical. For that reason I consider the current implementation a prototype and have plans to migrate to a more scalable system.

One possibility is to move towards using Abstract Syntax Trees (AST). The idea here would be to identify relevant snippets of source code or entire files that are candidates for modification, parse them into an AST, and then use one of several methods from converting AST back to source code.

There would be still be plenty of complexity still even with this dramatic simplification such as comments, preservation of formatting (support for configurable formatters through which the regenerated source code could be passed?), functions in other files, global variables, etc.