search

rajasekarv / vega

A new arguably faster implementation of Apache Spark from scratch in Rust
Apache License 2.0
2.23k stars 205 forks source link

Create basic integration tests from the examples #12

Closed gzsombor closed 4 years ago

gzsombor commented 4 years ago

Originally, I tried to put everything in one file, however I've encountered a couple of problems - so yes, tests could detect bugs :wink: - which caused panics, and dead locks, due to the shared global ShuffleManager and the http server, which tries to bind to the same port, and blocks indefinitely. I will try to fix these issues as well, but I thought creating a baseline tests for the current, supported functionality.

rajasekarv commented 4 years ago

It can easily happen as cargo tests are run in parallel and I am basically using the same port for map output tracker master server, throughout multiple runs, which should be changed(I will fix it soon. It is just a bit convenient when I am testing on multiple machines without bothering too much with configurations). So what you can do for now is create a SparkContext as a static variable and use that for all tests within a file which should prevent those deadlocks and panics.

rajasekarv commented 4 years ago

In addition, maybe there is some problem due to args passed along with cargo test. I will fix this now and commit it(changes will be in context.rs and env.rs). You can pull that change and test.

gzsombor commented 4 years ago

Thanks, with the recent changes this will work:

cargo test -- --test-threads=1
rajasekarv commented 4 years ago

Thanks, @gzsombor. Yeah, either it has to be run with one thread or a global static Spark Context needs to be shared between multiple tests in a file.