apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine
https://datafusion.apache.org/ballista
Apache License 2.0
1.55k stars 197 forks source link

Expose more configuration options for executor #1098

Closed milenkovicm closed 3 weeks ago

milenkovicm commented 4 weeks ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

On the road to library transition I would like to expose more configuration (DataFusion specific) which user can tune. This would enable us to remove hard coded object stores, codecs, configuration extension, functions ...

Describe the solution you'd like

I would like to expose additional method in executor/src/standalone.rs:

pub async fn new_standalone_executor_from_state<
    T: 'static + AsLogicalPlan,
    U: 'static + AsExecutionPlan,
>(
    scheduler: SchedulerGrpcClient<Channel>,
    concurrent_tasks: usize,
    session_state: &SessionState,
) -> Result<()> {

And optional parameter in executor/src/executor_process.rs:

pub struct ExecutorProcessConfig {
    ...
    /// SessionState which will be used to configure executor
    /// if not provided system will configure itself from 
    /// sensible defaults
    pub session_state: Option<SessionState>,
}

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context

depends on #1096