search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Set up CI build (Travis) for testing #8

Closed ash211 closed 7 years ago

ash211 commented 7 years ago

Always good to have CI running for large projects! Should include both the unit tests and the integration tests (using minikube).

@erikerlandson suggests Sean has experience here and could help get this going.

ssuchter commented 7 years ago

Yep, someone over here will take this one. I'll report back.

ash211 commented 7 years ago

@ssuchter any progress on this? We're starting to have a few smaller commits a day and CI assurance is always nice :)

ssuchter commented 7 years ago

Not yet, I didn’t realize the urgency - we were going to get to it next week. The current Spark Jenkins is hosted by Berkeley, so one just add things to it. We were going to look into using either Travis or us hosting a Jenkins node. Will looking into this next week be ok?

Sean

On January 11, 2017 at 4:05:57 PM, Andrew Ash (notifications@github.com) wrote:

@ssuchter https://github.com/ssuchter any progress on this? We're starting to have a few smaller commits a day and CI assurance is always nice :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache-spark-on-k8s/spark/issues/8#issuecomment-272035833, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2E27lt-7vY5p5J2j95JyhP1_8LiPxkks5rRW5igaJpZM4LfxWS .

ssuchter commented 7 years ago

We’re looking into it now - update on ETA tomorrow.

Sean

On January 11, 2017 at 4:22:03 PM, Sean Suchter (sean@suchter.com) wrote:

Not yet, I didn’t realize the urgency - we were going to get to it next week. The current Spark Jenkins is hosted by Berkeley, so one just add things to it. We were going to look into using either Travis or us hosting a Jenkins node. Will looking into this next week be ok?

Sean

On January 11, 2017 at 4:05:57 PM, Andrew Ash (notifications@github.com) wrote:

@ssuchter https://github.com/ssuchter any progress on this? We're starting to have a few smaller commits a day and CI assurance is always nice :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache-spark-on-k8s/spark/issues/8#issuecomment-272035833, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2E27lt-7vY5p5J2j95JyhP1_8LiPxkks5rRW5igaJpZM4LfxWS .

ssuchter commented 7 years ago

Heh, fixing the last comment. We did some experimentation last night, and it seems like Travis CI will work well for this. Working on a doc and/or actually setting it up.

ssuchter commented 7 years ago

Ok, I've got Travis working on my fork - once I learned how to do it, was very simple.

https://github.com/pepperdata/ssuchter-test-spark and https://travis-ci.org/pepperdata/ssuchter-test-spark

I need some minor permissions on the apache-spark-on-k8s project to do it (not code submission, but permissions about creating integration and adding comments to issues, because Travis will want to do it.) I can either describe it to one of you, or I can do it. For now, I'll just send a request for the permissions I need, that might be the simpler way.

foxish commented 7 years ago

Approved access! Thanks @ssuchter.

ssuchter commented 7 years ago

Grr. Apparently I need to have more access to set this up. I think ultimately we will want a role account for this, but maybe temporarily could someone make my account an "admin" so Travis setup will work? I promise that I won't do anything else and I want to revoke that access later.

See attached image that has the error message:

apache_spark_on_kubernetes_-_profile_-_travis_ci

ssuchter commented 7 years ago

I didn't notice this before because on our organization, I'm already an admin. :/

ssuchter commented 7 years ago

Ok, I turned Travis CI on. I've got a couple meetings, but in a little bit I'll try making a dummy pull request to try it out.

foxish commented 7 years ago

Sounds good!

ssuchter commented 7 years ago

I should note that this is not yet doing anything beyond what the builtin .travis.yml does. We probably need to modify that, and also we will have to work on a minikube based test, as imagined earlier. That's a little more work than just the raw setup.

ssuchter commented 7 years ago

BTW, see Travis here: https://travis-ci.org/apache-spark-on-k8s/spark

ssuchter commented 7 years ago

Yay, it's working (well at least it's running the tests... we'll see if they succeed soon-ish):

added_service_name_as_prefix_to_executor_pods_by_foxish_ _pull_request__14_ _apache-spark-on-k8s_spark

kimoonkim commented 7 years ago

Hi, I am working with @ssuchter for the travis CI setup. The current .travis.yml file does not use the kubernetes build profile.

I just sent a pull request #21 with a fix adding the profile.

ash211 commented 7 years ago

This seems to be working and is catching issues -- @kimoonkim I think we're good to close this? Anything more we ought to do immediately with the .travis.yml file?

kimoonkim commented 7 years ago

@ash211 Yes. I think we can close this. Sean and I started looking at integration tests, but we should open a new issue when we get it going and have something to discuss. Thanks.