Feature: Position/Keyword Args with SQL

[sjrusso8]

Description

Implement the ability to use positional/keyword args with sql. Because of the differences between python and rust, the function arguments need to be clearly implemented.

The pyspark process for sql allows for literals and dataframes to be in one argument. However, rust probably won't take to kindly to that input arg. If a user passes in a DataFrame it will need to be handled with a SubqueryAlias and if it's a literal it will be passed in as either a positional or a keyword argument.

We might want to only allow for to inputs parameters are options. Something like this?

sql<T: ToLiteral>(self, sql_query: &str, col_args: Option<HashMap<String, T>>, df_args: Option<HashMap<String, DataFrame>>) -> DataFrame 

This could allow a user to do these variations.

spark.sql("SELECT * FROM table", None, None).await?;

let df = spark.range(...);

// create the hashmap

spark.sql("SELECT * FROM {df}" None, Some(df_hashmap)).await?;

let col = "name";

// create the hashmap

spark.sql("SELECT {col} FROM {df}", Some(col_hashmap), Some(df_hashmap)).await?;

Or should positional SQL be a completely different method all together? like sql_params? So that a user doesn't need to fuss with adding None x2 to all their sql statements.

[hntd187]

The majority of the stuff is probably covered in a format! similar macro, or format_args! some of the other ones with direct DF objects in place may get a bit strange there, but the simple value ones should be easy to do with format and nothing else.

[sjrusso8]

I’ve haven’t had the time to code it up, but I’m interested in creating a macro like sql! which will handle this. But it’ll also perform similar to what you’d find with the sqlx query! macro that performs validation at compile time.

It would be interesting to have your sql transformation validated by the compiler. I feel like it could be a helpful feature.

[abrassel]

Can I take this? I want to try to make the sql! macro. @sjrusso8

[sjrusso8]

@abrassel go for it!