Automated Testing

[hallister]

In order to streamline building, we should probably look at automated testing. One of my favorites is http://travisci.org. The biggest upside to travis is it integrates with Github, so when pull requests are opened, the tests are run automatically and the pull request is updated with the status of the run tests (pass/fail).

Any other suggestions?

[andyleejordan]

I highly suggest using Jenkins, as we can have full control over it. I already have an instance running here, and can give users control as we decide. We won't have to depend on TravisCI, and Jenkins is highly configurable. It has Github integration among many other plugins (go ahead and make a user, post here and I'll give you permissions to view these).

It can be setup to run automatically on pushes to Github, run unit tests, run pylint/flake8/pep8 syntax checkers/validators, build packages, and literally anything we wish to script. It can then trigger further builds and notify the appropriate parties in whatever means they'd like.

BTW it's been my paid professional job over the last year at SEL to mange a CI environment like this (using Jenkins, mostly Mercurial and some Git, and we looked into TeamCity). We also used CodeCollaborator for code reviews, but I want to look into some others that I can host (such as ReviewBoard). Also, I'm donating the use of my server for our project. It's a Puppet managed VPS that I can add anything we'd like to. For ~$10 we can get our own CS383 FITG domain to point to here to, instead of schwartzmeyer.com (thanks to the magic of nginx virtualhosts and proxies).

[hallister]

Well, Jenkins or TravisCI are still dependencies either way :smile:.

That said, I think we may be over doing this. Having Github, a Test Server and a Code Review server is a fairly disjointed work flow for a project of 16 weeks with 15 people.

[andyleejordan]

Oh believe me, we could overdo this if we wanted, but regardless we're going to want Github for hosting and issues, a test server just makes our lives easier, and will be transparent to most of the group (i.e. overall it saves everyone time), and a full-fledged code-review server would like be much easier than trying to do it on Github.

If you're worried about server setup and maintenance, don't be. I already do it anyway.

[hallister]

I'm for a Test Server, but the reason I like TravisCI is because we don't have to worry about hosting it. With Jenkins we do. Not to mention, a single tool that does it all is kind of the point of Github.

[andyleejordan]

I'm saying you don't have to worry about hosting it (I already host it myself; it won't be an issue for others, it's kinda just what I do as a devops guy), but you do have to worry about TravisCI as a host. I've run into issues with them a few times on other projects.

Yeah it'd be nice to have a pre-rolled all done solution, and Github hits most of it. I'd like to try out a few code review servers, but looking at our options (and even I hate CodeCollaborator) Github's basic code review may be best.

Speaking of options:

• Phabricator (which I can setup using this Puppet modle, but I'll need to update it to use nginx instead of apache, something I can do this weekend.
• Review Board probably just write my own Puppet manifest for this
• Gerrit there's an out-of-date module for this

Note: I haven't used any of those three options, they're simply available. I personally (as a software developer) loathe code reviews, because come on, they go on and on sometimes.

[bcumber]

Here's my question: Which one is easier? Basically, I know that personally for this project I need to learn python, LaTeX, how to use git/github, and probably more things that I don't know about yet. Obviously this is probably true for more than just a few of our group and each of us also has other responsibilities outside of this class. So which one is easier to learn in the short amount of time that we have?

[andyleejordan]

I'd say using Jenkins is already easier as I have used it for the last year at my job for these exact purposes. I already have it hosted and will take responsibility for maintaining it and the test jobs.

As for the review process, probably just using Github. (In which case, Pull Requests may actually become handy, albeit slow).

[hallister]

@b-cumber That's a tough question. But here's what I can tell you about TravisCI:

It requires a small configuration

Putting this at the root of the repository will run all of our tests against all the listed python versions:

language: python
python:
  - "3.1"
  - "3.2"
  - "3.3"
# command to install dependencies
script: python tests/start.py
branches:
only:
  - master

It integrates with Github Pull Requests

Should we decide to use Pull Requests TravisCI integrates directly with them:

As you can see it says right on your Pull Request that the tests have passed. This one shows a failed build:

It has a nice user interface to track down failed tests

https://travis-ci.org/hall5714/angular-strap/jobs/9570066

Here's a test I ran that failed. It shows all of the output of the console, making failed tests easy. You can even follow it while it's testing.

We can implement TravisBot

JenkinsComments (something similar for Jenkins) for example, only tests the fork. TravisBot actually tests the merged version of the fork, which guarantees no failed tests on the merged version.

All in all, neither of them are bad, but frankly TravisCI is the best if you are using a Github PR workflow. Any other workflow and the benefits of TravisCI and Jenkins are about on par... although Jenkins has more plugins and TravisCI has better User Interface.

[hallister]

@b-cumber I tried to find a good comparison between the two for you, but like many things in software engineering it's a lot of fanboism on both sides.

All I can tell you is that tools like Jenkins are incredibly useful and incredibly powerful, but unnecessary for a project like this. This is a small project over the course of a couple of months. If we were building a 3 year project complete with nightly's I'd say Jenkins wins this all day. But since we are, I think the best option is the tool that does the job in the easiest way possible. That's Travis.

[bcumber]

So here is my next question for @andschwa and anyone else I guess. Typically it's a bad idea to put all of our eggs in one basket. If there were ever a time where something happened with Jenkins or we needed access to it and you are unavailable, would we be able to function without you? Because while you say that you

... will take responsibility for maintaining it ...

we don't necessarily want all that responsibility to be with a single person. Forgive my ignorance if this isn't even a problem lol.

[hallister]

@b-cumber Not ignorant at all Ben, and I point that I missed. Obviously TravisCI is remotely hosted, so if anything happens it's them that deals with it... and given that thousands of projects rely on them, I doubt that will be an issue. I haven't been able to find a hosted Jenkins CI that isn't paid, but perhaps there is something similar for Jenkins.

[andyleejordan]

@hall5714 Also, unless we open-source and license our project, we can't legally use Travis CI. I've seen it go down very frequently.

@b-cumber I pride myself on high availability of the servers I maintain, but if something were to happen that would require fixing, I can provide a user account on my VPS for another user to fix stuff if needed. If it's a direct issue with a Jenkins job, I expect other testers will want Jenkins accounts if they want to maintain any jobs without me (which I've already mentioned I'll provide, just go make one and tell me the permissions you want).

Personally, I much prefer testing on each push to a repository, rather than on the much less frequent Pull Requests, as this provides faster and almost-immediate feedback if a changeset caused the unit tests to fail. Correct me if I'm wrong, but I don't believe Travis CI provides this.

[hallister]

@andschwa TravisCI is free if you have an public repository, it costs money if you have a private repository. This is not a private repository (because that would cost us $25/month).

I'm a little confused on your last statement. If we use the PR workflow model, then there won't be pushes to the master branch, there will only be PR's. So in that case, using TravisCI is necessary, because we need to know if the tests pass before the merge, not after.

But to answer your question, TravisCI runs the build on all Github hooks, including Push and PR.

[andyleejordan]

If this is a public repository we need to add a license ASAP. We should be in the safe harbor for use of FITG as this is an academic project, but we should check with Dr. J. Private repositories are provided for free for academic uses.

I'm now confused with your statement, there isn't such a thing as 'pushing to the master branch'. You push to a repository, regardless of which branch you've been committing on. A PR request is started from a push to a Github fork (none of this is git, it's Github's system), which can still trigger a Jenkins or a TravisCI build, we'll just have to manage more repositories, which frankly is going to be more complex. You don't want to test before a merge, you need to merge master into the feature branch and test, if it passes, merge into master. If you test your feature branch without merging master into it, you're not actually testing compatibility of the code.

That's handy and gets rid of that worry.

[hallister]

I'm pretty familiar with this workflow actually. And my point was, with the PR model we won't be pushing to the organizational branch where the tests occur.

"you need to merge master into the feature branch and test, if it passes, merge into master."

That's a lengthy process. TravisCI tests the fork and if it passes let's you know. If after the merge a different test breaks, then the build status will show that it failed. Regardless, it should be an pretty rare event that merging 2 branches, both with all tests passing will result in a failed test post-merge.

I suppose we could add a dev or feature branch to the organization which we merge into and use the build status from there, but that seems like adding steps we don't need.

"If you test your feature branch without merging master into it, you're not actually testing compatibility of the code."

Why? If the master is passing and the fork is passing and the master and fork can be fast-foward merged, how could a test fail after the merge?

[andyleejordan]

There's still no such thing as pushing to a branch, so I'm unsure what that means.

That's not a lengthy process at all, it's automated, and in fact that is how tests are done. I don't know how you'd test otherwise.

I was expecting we'd be using an integration branch, and it's practically mandatory to work on features using feature branches. Have you read the 'must reads' section in this issue: https://github.com/ui-cs383/Freedom-Galaxy/issues/10? Specifically, the branching model article.

I think by master here you mean 'upstream repository', i.e. origin/master. Yes, if the downstream repository's (i.e. fork's) master branch is a direct descendant of the upstream's master branch, then there would not be a failed merge. Regardless, in order for this to happen, you must have merged upstream's master branch into the integration branch to be testing the merge. When testing feature branches, you'd merge the integration branch into them.

Cheers,

Andrew Schwartzmeyer

On Sep 14, 2013, at 8:49 PM, Justin Hall notifications@github.com wrote:

I'm pretty familiar with this workflow actually. And my point was, with the PR model we won't be pushing to the organizational branch where the tests occur.

"you need to merge master into the feature branch and test, if it passes, merge into master."

That's a lengthy process. TravisCI tests the fork and if it passes let's you know. If after the merge a different test breaks, then the build status will show that it failed. Regardless, it should be an pretty rare event that merging 2 branches, both with all tests passing will result in a failed test post-merge.

I suppose we could add a dev or feature branch to the organization which we merge into and use the build status from there, but that seems like adding steps we don't need.

"If you test your feature branch without merging master into it, you're not actually testing compatibility of the code."

Why? If the master is passing and the fork is passing and the master and fork can be fast-foward merged, how could a test fail after the merge?

— Reply to this email directly or view it on GitHub.

[hallister]

When someone says "push to master" what they usually mean is:

$ git push -u origin master

Which would push your local master branch to the remote "origin's" master branch. "Pushing to master" was may shortcut way of referring to pushing to a remote branch from a local branch.

And I'm aware of how the develop branch model works. It's one model (a good one). However, many projects use the release-only model. Meaning their master branch is their development branch, and they branch a release branch for shipping.

This is part of the model I suggested. If we don't use it, I agree merging directly into master is a horrible idea. But if we do use the model I suggested, merging into master is a part of the workflow (because the tests have already been run and the code has already been reviewed).

[andyleejordan]

I'd never heard it used that way, so I misinterpreted it.

Just to be clear on which model we're using, I've been under the assumption of master containing only releases (using Semantic Versioning), and an integration/development branch for testing and putting together the next working version, as this is what I thought had been discussed. If we're switching to a release branch model, then essentially we're just moving some names around, but that should be stated.

[hallister]

@andschwa I'm not sure we decided either way. I would assume the model we pick would dictate that. The Push/Branch model you suggested would clearly benefit from the branching you describe. But I think the PR/Fork model tends to use release branch models more often.

[andyleejordan]

I think both models fit equally well, it really is just a set of names. There are two branches: the master (in the role of release or development) branch, and then either a release or development branch, whichever is missing. Same model, two naming schemes. But it's just gotta be set before we begin, as we don't have a policy yet. Votes?

Cheers,

Andrew Schwartzmeyer

On Sep 14, 2013, at 10:19 PM, Justin Hall notifications@github.com wrote:

@andschwa I'm not sure we decided either way. I would assume the model we pick would dictate that. The Push/Branch model you suggested would clearly benefit from the branching you describe. But I think the PR/Fork model tends to use release branch models more often.

— Reply to this email directly or view it on GitHub.

[hallister]

@andschwa That brings up a good point. We really need to start getting these things decided.

Perhaps we need to start setting dates for voting? If you don't vote by X date then we assume neutral. Anyone have thoughts/opinions on this? We REALLY need to have some of this stuff worked out.

[andyleejordan]

@hall5714 Yeah, does anyone in @ui-cs383 have business management experience? We have some setting up / organizational things to do.

I think an issue can be posted for a vote, and then closes for voting in 4 days would be fair, as it would ensure always one class period occurs. If a two-thirds (i.e. 12) vote is reached (as it's public, not blind), the vote closes so we can get on with things.

[hallister]

Closing per last meeting. Decided TravisCI due to PR integration.

[andyleejordan]

Just FYI that wasn't actually discussed. PR integration exists fine with both Jenkins and TravisCI, but if we have problems with TravisCI, I'm not the one managing it, so I won't be able to fix it.

Cheers,

Andrew Schwartzmeyer

On Sep 19, 2013, at 11:10 AM, Justin Hall notifications@github.com wrote:

Closing per last meeting. Decided TravisCI due to PR integration.

— Reply to this email directly or view it on GitHub.