Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I would like to propose to trim down BallistaConfig, make SessionConfig as main way to configure Ballista and make BallistaConfig just a part of it, with ballista specific configurations
ballista specific configuration, which should be preserved:
+-------------------------------------------------------------------------+---------------------------+
| name | value |
+-------------------------------------------------------------------------+---------------------------+
| ballista.grpc_client_max_message_size | 16777216 |
| ballista.job.name | |
+-------------------------------------------------------------------------+---------------------------+
we can see that configuration map 1 to 1 to datafusion configuration
Note: datafusion.optimizer.enable_round_robin_repartition has to be false false ;
Describe alternatives you've considered
keep BallistaConfig but at the moment i see no benefit of keeping it or scenarios which SessionConfig cant support
Additional context
As it stands with introduction of SessionContextExt, BallistaConfig has been removed from all public interfaces, with this change BallistaConfig will probably be removed from most internal interfaces as well.
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I would like to propose to trim down
BallistaConfig
, makeSessionConfig
as main way to configure Ballista and makeBallistaConfig
just a part of it, with ballista specific configurationsDescribe the solution you'd like
merge following configuration:
If we check
/ballista/scheduler/src/state/session_manager.rs
ballista specific configuration, which should be preserved:
we can see that configuration map 1 to 1 to datafusion configuration
Note:
datafusion.optimizer.enable_round_robin_repartition
has to be falsefalse
;Describe alternatives you've considered
keep
BallistaConfig
but at the moment i see no benefit of keeping it or scenarios whichSessionConfig
cant supportAdditional context
As it stands with introduction of
SessionContextExt
,BallistaConfig
has been removed from all public interfaces, with this changeBallistaConfig
will probably be removed from most internal interfaces as well.