apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine
https://datafusion.apache.org/ballista
Apache License 2.0
1.39k stars 181 forks source link

Expose the lower level scheduler API #888

Open feribg opened 9 months ago

feribg commented 9 months ago

I was reading through examples and source but there doesn't seem to be a very obvious way to interact with the scheduler.

Since currently there dont seem to be any good distributed computing libraries in the rust ecosystem and ray has no plans to offer official rust support, it would be a very nice addition to be able to interact with the underlying tasks in 2 main ways:

maybe there is already a straightforward way to do that in which case im happy to close this and contribute to the docs.