Adds dynamic allocation tests

lightbend / mesos-spark-integration-tests

Mesos Integration Tests on Docker/Ec2

16 stars 9 forks source link

Adds dynamic allocation tests #46

Closed skyluc closed 8 years ago

skyluc commented 8 years ago

Also adds starting the shuffle service on the docker cluster

skonto commented 8 years ago

Any doc update needed?

dragos commented 8 years ago

What's the status of this PR? Can we merge it?

skonto commented 8 years ago

Have not tested it yet if it runs without issues i am ok...

skonto commented 8 years ago

I just got this, first time trying to test pr on linux with mesos 0.26 and spark master (downgraded, 0.27 is buggy):

ClusterModeSpec:

simple sum in fine-grained mode
simple collect in fine-grained mode
simple count in coarse-grained mode !!! IGNORED !!!
spark.cores.max property should be honored in coarse grain mode
simple count in fine-grain mode with role
simple count in coarse-grained mode with role !!! IGNORED !!!
dynamic allocation, in coarse grain mode * FAILED * The test did not complete within the specified 300 seconds time limit. (TimeLimitedTests.scala:137)

ERROR: 1 test failed Is there something i should enable?

skyluc commented 8 years ago

Are you running against the current master? The fix for the initial number of executors has been merge. I need to see what that changes.

skonto commented 8 years ago

I will run again if not updated. Was not sure what version to check with anyway... github is not made for testing we have to add comments for spark testing version when reviewing a pr when that is appropriate, because someone looking at this could not have imagined it :). Also i made a question above about documentation any comments?

skyluc commented 8 years ago

I have updated the show_help function, I have not see any other location with the description of the command line parameters for the docker script.

skonto commented 8 years ago

Ok great thnx just to be sure.

skonto commented 8 years ago

I run with latest: last git log 7/2 previous was older. Both tests fail in client and cluster mode. I didnt see anything strange in the logs. i could attach them here as well if that helps, they look similar with the previous.

I0208 16:35:43.464270 25647 sched.cpp:1043] Stopping framework '76c03c77-e881-4a26-9b89-1282f1f6e85b-0004'

dynamic allocation, in coarse grain mode * FAILED * The test did not complete within the specified 30 seconds time limit. (DynamicAllocationSpec.scala:29)
dynamic allocation, in coarse grain mode * FAILED * The test did not complete within the specified 30 seconds time limit. (DynamicAllocationSpec.scala:29)

ERROR: 2 tests failed

skyluc commented 8 years ago

I'm seeing the same errors. I should have a fix shortly.

skonto commented 8 years ago

I tested it runs fine. What was the problem you fixed? LGTM i will merge it.