apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
5.87k stars 1.11k forks source link

What is the easiest way to turn statements into JSON? #6227

Open armanm opened 1 year ago

armanm commented 1 year ago

I know very little Rust so apologies for noobe question. I'm using DataFusion through Elixir and want to analyse user statements before passing them along to actually perform the query. How do I convert the output of parser::DFParser::parse_sql into JSON?

parser::DFParser::parse_sql("
    CREATE EXTERNAL TABLE foo(c1 int) STORED AS CSV LOCATION 'foo.csv';
    select * from foo;
    DESCRIBE table;
    INSERT INTO users (ss) values ('sdsd');
  ")

I noticed the underlying sqlparser crate supports serde trait for statements but from what I see datafusion-sql adds support for a couple more statements (CREATE EXTERNAL... and DESCRIBE ...) and that crate doesn't support serde.

What are some options for me?

alamb commented 1 year ago

Can you start with just supporting Statement::Statement?

So something like (untested)

let statement = parser::DFParser::parse_sql("
    CREATE EXTERNAL TABLE foo(c1 int) STORED AS CSV LOCATION 'foo.csv';
    select * from foo;
    DESCRIBE table;
    INSERT INTO users (ss) values ('sdsd');
  ")

let json = match statement {
   Statement::Statement(sql_statement) =>  serde_json::to_string_pretty(sql_statement).unwrap()
   Statement::CreateExternalTable(..) => "{}".to_string(),
   Statement::DescribeTableStmt(..) => "{}".to_string(),
}

That might be enough to get you unblocked -- then perhaps we can add serde support to https://docs.rs/datafusion-sql/23.0.0/datafusion_sql/parser/enum.Statement.html in DataFusion directly

armanm commented 1 year ago

Thank you for that example. Although Statement::Statement is useful, I'm mainly interested in the CREATE EXTERNAL TABLE statements because I intend to conditionally block those specifically 😓

That might be enough to get you unblocked -- then perhaps we can add serde support to https://docs.rs/datafusion-sql/23.0.0/datafusion_sql/parser/enum.Statement.html in DataFusion directly

That sould be the ultimate help but your suggestion does give me a path forward so I'm very grateful 🙏