Support running integration tests in CircleCI / TravisCI

[ash211]

I was chatting with @mccheah and @jlz27 about what it would take for us to run integration tests in CircleCI, and Jason has a setup that works, minus one problem.

Roughly the setup is to use minikube in the --vm-driver=none mode and modify docker bridge settings so any docker images running in the circle container (Spark the repo doesn't have this, but we have many internal repos using docker-compose for integration testing) can communicate with the launched kubernetes pods.

The catch though is that kubelet attempts to modify the kernel settings of the host it runs on to fit its preferences, but that doesn't work in CircleCI. See https://github.com/kubernetes/kubernetes/issues/50110 for details of that.

The benefits of getting this working would be:

• another option for how to run the test as we upstream this repo into apache/spark
• improve parallelism / throughput vs the minikube on jenkins setup: https://github.com/apache-spark-on-k8s/spark/issues/394 and https://github.com/apache-spark-on-k8s/spark/issues/280
• reduce need to poke minikube if things go wrong (since circle/travis start fresh on every build) https://github.com/apache-spark-on-k8s/spark/issues/279
• and personally, this would make it much easier for us to run the k8s tests on the Palantir fork of Spark at https://github.com/palantir/spark and we'd be more able to provide early feedback on interaction breaks between Apache master and the k8s branch-2.2-kubernetes until we upstream the k8s code

[mccheah]

I think the biggest benefit actually is just to remove the need to install a virtual machine on the upstream Spark's Jenkins nodes. Running Minikube on bare metal is much lighter weight than running Minikube inside a virtual layer.

[dimberman]

+1 this would also be valuable for the airflow-kubernetes integration tests.

[shaneknapp]

i have a worker set up w/minikube and kvm that seems to "work as intended". i'd really like to start testing changes like this against that worker as soon as we can.

[shaneknapp]

btw, the --vm-driver=none option is a show-stopper for the amplab/riselab/open source spark build system. it requires that the UID have root privs to many parts of the worker's system and is a potentially large security hole.