Closed gzsombor closed 4 years ago
It can easily happen as cargo tests are run in parallel and I am basically using the same port for map output tracker master server, throughout multiple runs, which should be changed(I will fix it soon. It is just a bit convenient when I am testing on multiple machines without bothering too much with configurations). So what you can do for now is create a SparkContext as a static variable and use that for all tests within a file which should prevent those deadlocks and panics.
In addition, maybe there is some problem due to args passed along with cargo test. I will fix this now and commit it(changes will be in context.rs and env.rs). You can pull that change and test.
Thanks, with the recent changes this will work:
cargo test -- --test-threads=1
Thanks, @gzsombor. Yeah, either it has to be run with one thread or a global static Spark Context needs to be shared between multiple tests in a file.
Originally, I tried to put everything in one file, however I've encountered a couple of problems - so yes, tests could detect bugs :wink: - which caused panics, and dead locks, due to the shared global ShuffleManager and the http server, which tries to bind to the same port, and blocks indefinitely. I will try to fix these issues as well, but I thought creating a baseline tests for the current, supported functionality.