search

apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine
https://datafusion.apache.org/ballista
Apache License 2.0
1.39k stars 181 forks source link

CI should run Ballista integration tests #24

Open andygrove opened 3 years ago

andygrove commented 3 years ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. We are seeing PRs merged that cause regressions in Ballista because we currently rely on manual integration testing.

Describe the solution you'd like CI should run the integration tests:

./dev/integration-tests.sh

Describe alternatives you've considered None

Additional context None

alamb commented 2 years ago

The lack of ballista coverage in CI came up again with @gaojun2048 @liukun4515 and @mingmwang on https://github.com/apache/arrow-datafusion/pull/2070#discussion_r834104400

liukun4515 commented 2 years ago

I can take this task in the Q2

houqp commented 2 years ago

Thank you @liukun4515 ! In case you missed it, the setup I had in https://github.com/apache/arrow-datafusion/pull/688 already works and was only paused pending optimizations to reduce cache size. I imagine you might be able to reuse some of the setup there when you start working on this this task.

andygrove commented 2 years ago

Should we consider running the integration tests in CI without creating docker images? Just run the native binaries for scheduler and executor?