Open wangkuiyi opened 4 years ago
The name sqlflow
may conflict with current pysqlflow command-line client: https://github.com/sql-machine-learning/pysqlflow/blob/develop/setup.py#L73
@tonyyang-svail: I can think of two workarounds:
sqlflowcmd
. Python client sqlflow
.sqlflow
. Python client pysqlflow
.sqlflow -codegen [tensorflow|xgboost] -db maxcompute < [a.json|a.sql] > a.py
->
sqlflow -codegen [tensorflow|xgboost] -db maxcompute -engine [pai_tf | elasticdl | alps | tensorflow | xgboost] < [a.json|a.sql] > a.py
Converts an IR of only one statement with extended syntax into a Python submitter program.
Maybe we don't need to convert IR into submitter program, instead of a submitter program accepts an IR as an argument: python sqlflow_submitter.tensorflow.train < ir.json
. So that we can implement a couler step function like:
couler.sqlflow.run(cmd='echo "SELECT ... TO TRAIN ... | sqlflow -parse | python -m sqlflow_submitter.tensorflow.train "')
sqlflow -codegen [tensorflow|xgboost] -db maxcompute < [a.json|a.sql] > a.py
I don't think we need to specify the [tensorflow|xgboost]
.
Because this information can be derived from a.sql
Let us merge the SQLFlow server and the REPL into a single binary, sqlflow.
@wangkuiyi We still need a gRPC server though.
Let us merge the SQLFlow server and the REPL into a single binary,
sqlflow
.This proposed merging is due to a decomposition of the translation process by SQLFlow.
Some steps of the generated workflow handle SQL statements with SQLFlow extended syntax. Such a step contains multiple steps:
All the above bullets are about kind of translation. We could use a single command-line tool to do all these translations.
sqlflow -parse < a.sql > a.json
Parse a SQL program (zero, one, or more statements) into IR presented as a JSON file.
sqlflow -codegen couler < [a.json|a.sql] > a.couler
Converts a SQL program or an IR into a Couler program`.
couler < a.couler > a.yaml
Converts a Couler program into an Argo YAML file.
sqlflow -codegen [tensorflow|xgboost] -db maxcompute < [a.json|a.sql] > a.py
Converts an IR of only one statement with extended syntax into a Python submitter program.
python a.py
Run the generated submitter program.