Shippable as alternative to jenkins and travis?

bgruening commented 9 years ago

This seems to be like a nice alternative to get galaxy in docker preinstalled and test tools with planemo. Since we have the information from tool_dependencies.xml we could even install packages from the TS before testing tools.

https://www.shippable.com/

@erasche I would appreciate your opinion as expert in this field!

hexylena commented 9 years ago

Ooh, always love a new CI competitor. (Btw, we can use docker in Jenkins, I just haven't gone to the trouble of configuring it.) I'll read up on shippable a little :)

Since we have the information from tool_dependencies.xml we could even install packages from the TS before testing tools.

Oh, how do you envision this happening? Shed2tap or similar?

bgruening commented 9 years ago

Eric;

no concrete plan, but I could imagine something like this:

we already have a TS Docker image
now we import all IUC, devteam repositories (build this image every 2 days)
import the tool/package into our TS Docker image
spin up Galaxy and install repository
use planemo to test the tool

hexylena commented 9 years ago

(Sorry for rambling, I just woke up...)

Not a bad plan at all! Having not dug into shippable very much I'm still thinking in terms of Jenkins....

IUC+devteam sounds good, we can call it "ToolShed Core" or something, and build that image (locally?) as part of the PR testing phase for both repos (I plan to add jenkins testing for tools-devteam)

I've thought about it a bit, and I don't think this'll be a separate job, I don't think we can take advantage of Jenkins job dependencies here. I think this has to be part of the build/test phase for each repository. Consider the following:

PR comes in with package + tool
need to rebuild TS image
only then tool testing will pass.

That makes me a bit sad, as I really love having job dependencies in Jenkins. Oh well. We need to develop a Makefile of some sort for handling the build/testing process. I'll probably do that in https://github.com/galaxy-iuc/jenkins-testing as I'd like it to be easier for others to set this up.

bgruening commented 9 years ago

Yeah, on the other side I would like to use such services integrated in github. Is there a free Jenkins hosting service, like travis?

I don't think we need to rebuild the image for every PR. We don't change our packages so frequently and most of them should be included. So this can be pre-calculated every few days from docker-hub.

hexylena commented 9 years ago

Sure, using services "properly" integrated into github would be nice...but I'm not aware of a hosted Jenkins solution, or really what benefit we'd get out of it over the VM I've dedicated for IUC use.

If we can get the TS/docker/planemo installing tools bits working, then I'll turn off "unstable" builds and builds will produce a hard failure if even one test fails. I'd like to see new tool PRs build and test successfully rather than have a delay of 2 days causing build failures. I don't mind the CPU overhead; that VM averages <5% utilisation. It's completely wasted! I want to test ALL the things just to get my money's worth out of the compute time allocated to it. :P

bgruening commented 9 years ago

I have just a bad conscience to use your money for this. It's not just IUC, it's devteam, galaxytools and probably galaxyp-tools. Do you really think there will be a delay for 2 days, I don't think so. But importing all IUC/devteam repos on demand will also work, if the delay turns out to be a problem.

hexylena commented 9 years ago

Nah, don't worry. I've been happy with Digital ocean so far, and I use that VM for a couple of personal purposes as well. If someone ever gets around to setting this up elsewhere (i.e. research funded), I'm happy to move it then.

Well, here's an example:

Imagine that the PR is perfect the first time around and requires follow up commits.
the first set of tests will fail as the associated package isn't installed.
In <2 days I'll have to add another commit to trigger a rebuild, at which point the packages will be installed (from a PR branch? Via magic?), and the tests will pass.

This is in contrast to:

Perfect PR is opened
Build TS image from tools-iuc/packages_* + tools-devteam@master.
Test again a galaxy instance connected to that local TS image.
Tests pass the first time around.

I literally have more CPU time than I know what to do with. If it takes another 15 minutes to build that image, or if it bumps my CPU usage up to 10, 20, 30%, I really, really don't care compared to the overhead of not being able to accept a PR after the very first set of passing tests.

Btw, I'm happy to set up jenkins jobs for all of the repos you mentioned, I just want to streamline the process first and get the TS/package_ installation working first. Again, I have more than enough CPU time to accommodate this.

bgruening commented 9 years ago

An other possibility seems to be circleci: https://circleci.com/docs/docker This could be used to test galaxy-stable, but with a bit of magic also uploading/installing and testing of tools (based on galaxy-stable).

hexylena commented 9 years ago

Yeah, that is another alternative. Getting the "magic" figured out is the only hard part. There are plenty of free options for us once we get there :)

bgruening commented 9 years ago

I'm thinking about activating the TS inside of Galaxy Docker ... with this in place it should be doable.

hexylena commented 9 years ago

I was using smithcr's TS image to handle this, hadn't worked out the steps to:

find all package dependencies that are not in the current repository
mirror those dependencies from the MTS
upload all packages (shed_serve doesn't seem to do quite that)
the run shed tests.

bgruening commented 9 years ago

First one seems to be the hard one, on the other hand all packages should be owned by IUC ;) ...

hexylena commented 9 years ago

Oh what a nice world that would be :P

bgruening / galaxytools

Shippable as alternative to jenkins and travis? #98