sql-machine-learning / sqlflow

Brings SQL and AI together.
https://sqlflow.org
Apache License 2.0
5.09k stars 698 forks source link

Use AST style code to refine optimize codegen #2531

Open sneaxiy opened 4 years ago

sneaxiy commented 4 years ago

Currently, the optimize codegen needs to call Expr.ToTokens() to get expression tokens and do code generation.

Considering the following SQL statements:

SELECT * FROM xxx
TO MINIMUM SUM(product)
CONSTRAINT xxx
...

The objective SUM(product) needs to be translated to be sum([model.x[i] for i in model.x]), which is the required form of Pyomo objective function. In the current implementation, we split SUM(product) into ["SUM", "(", "product", ")"], and then replace "product" with "[model.x[i] for i in model.x]", which is not a gentle way.

A better option is to use AST package to analyze the objective function and use something like ast.Walk or ast.NodeTransformer to do the substitution. We need to refactor the optimize codegen after developing AST package of SQLFlow.

wangkuiyi commented 4 years ago

I realized that https://golang.org/pkg/go/ast/ contains types like those we have in the .y file. Let us move those types out from the .y file into a new package pkg/parser/ast.