Compulsory test in travis?

[sauloal]

Should all packages have a test suit created or have their testing compulsory run in travis?

E.g.: if the package contains a "make test", the travis should not only test the compilation but the "make test" routine also. if the package does not contain a test suit, a small test should be created and tested in travis

[ypriverol]

@sauloal I proposed this idea before, I think it should be mandatory a set of tests for every package with a proper directory to run them in travis. It will allow us: 1- Know if the package is useful and detect in travis problems with software. 2- Short tests about how to use the container.

I know other open-source projects Suggest the test but don't make them mandatory, from my point of view we should try to encourage them. Also for best practices of development, deployment and integration.

[bgruening]

@sauloal is this possible with travis? Can you run Docker containers in travis and build them? There are other CI tools that support Docker.

:+1: for making tests mandatory in some way :)

[ypriverol]

@bgruening docker integration with travis https://github.com/moul/travis-docker

[bgruening]

@ypriverol thanks, so many things to learn :)

[prvst]

i agree with having tests for the containers, but I have some concerns about it. There is a thin line between testing the container and testing the software inside. I don't thin we should test the software inside because I don't think that's our problem. If a software works or not, that's not something that we should test. We should only worry about the containers itself.

On Thu, Aug 20, 2015 at 5:28 AM Björn Grüning notifications@github.com wrote:

@ypriverol https://github.com/ypriverol thanks, so many things to learn :)

— Reply to this email directly or view it on GitHub https://github.com/BioDocker/biodocker/issues/15#issuecomment-132948755.

[sauloal]

@bgruening , I didn't know of any way to run docker directly on travis. apparently they have support for it now.

That said, @Leprevost , I think we have to test if the program runs. Often the dockerfile will download, compile and run. All of which can fail. Dockerhub webhooks, as far as I remember, won't email you to let you know the compilation failed. I agree we don't have to check if the results are correct, only if they pass their own test suit or if they do not crash when started.

This could be simplified by requesting a "test.sh" script along the dockerfile which could be run automatically using the "ONBUILD" directive

[ypriverol]

@Leprevost see this as bioconductor or other open-source project when you specify a set of tests to be future your tool is working. For example, if your docker container and source contains dependencies to other tools, and these dependencies where updated, then the container well fail and do we want a huge garbage hub of containers, we also "care" about the software inside the container, the availability and functional behaviour.

[sauloal]

@ypriverol , sorry I missed your former mention of this idea. either way, I think it is nice to have one "issue" to each discussion both for archival and referencing. "I remember we talked about it. which conversation was it?!?!"

[sauloal]

@ypriverol , I second your last comment

[sauloal]

Repeating my comment on #5 : @Leprevost & @ypriverol I've been digging. If we are setting webhooks to re-create the images upon commits and dockerhub will run the dockerfiles and report the result ( https://docs.docker.com/docker-hub/builds/#webhooks ) with full log and return status (just like travis). why do we need travis?

[bgruening]

@sauloal is this also working for PR? DockerHub is no CI it's a storage and it will email you if a build is failing.

[prvst]

I noticed that since the dockerhub upgraded I started getting Travis emails from their platform.

On Thu, Aug 20, 2015 at 2:45 PM Björn Grüning notifications@github.com wrote:

@sauloal https://github.com/sauloal is this also working for PR? DockerHub is no CI it's a storage and it will email you if a build is failing.

— Reply to this email directly or view it on GitHub https://github.com/BioDocker/biodocker/issues/15#issuecomment-133113730.

[sauloal]

@bgruening , yes, it also works for PR. dockerhub, with automatic builds, works exactly as travis and can also automatically send you a message on error exactly like travis.

unlike travis, you don't need to do the same work twice: once to make it work on docker, another to make it run on travis.

[sauloal]

http://ariya.ofilabs.com/2014/12/docker-and-phoenix-how-to-make-your-continuous-integration-more-awesome.html

[prvst]

I think we can leave travis and travis-alike out for now and focus on making the repositories functional and available to users. Because DockerHub also provide us with reports we are not going to be prejudiced by this for now

[sauloal]

agree.

[sauloal]

/cc @BioDocker/contributors hey, I've made a better version which only has the build information instead of all the image information: http://status-biodocker.rhcloud.com/

[ypriverol]

@BioDocker/contributors I strongly think we should made mandatory in the description the tag, Biodocker for all the containers. In the future if someone provides the containers in another hub folowing our specifications, we should be able to find them @sauloal

[sauloal]

sorry @ypriverol , i don't understand what you mean