[devel] continuous integration

[juju4]

• work in progress for CI config travis or kitchen. https://github.com/juju4/sift-saltstack/tree/devel https://travis-ci.org/juju4/sift-saltstack
• travis As a note, travis easily stall in my experience for unknown reason. kitchen docker works locally Note that saltstack return an incorrect exit code (https://github.com/saltstack/salt/issues/18510) so build is green even if not all tasks have succeed Mixing with docker and lxd is required as travis only support precise and trusty (https://github.com/travis-ci/travis-ci/issues/5821) As for why multiple provider (lxd vs docker), from a testing perspective, got different issues. as docker files are usually minimal, got more often dependencies issues. lxd is more similar to a VM or bare-metal. from a production perspective, docker is an application container (as said in official docs, 1 container, 1 process) while lxd is a system container and reflects better a real system/usage AFAIC

In the end, got it but it seems execution is too long for the maximum allowed by travis (50min bet it with docker or with lxd) Errors left were: https://travis-ci.org/juju4/sift-saltstack/builds/202262518#L4943 (regripper) https://travis-ci.org/juju4/sift-saltstack/builds/202262518#L5508 (wine) not the right order it seems. (docker run) https://travis-ci.org/juju4/sift-saltstack/builds/202285669 = probably mean sift as a unique salt play is too big...

• kitchen: fine locally. variable in travis
• Jenkinsfile (calling kitchen) for own CI server with kitchen or else

[ekristen]

Thanks @juju4 I'll take a look at what you put together. After seeing the initial stuff last week I went ahead and put tests on each individual salt state as it changes using Travis. You can see the current work now in https://github.com/sans-dfir/sift-saltstack

SaltStack has a retcode passthrough option that my scripts take advantage of.

As for the timelimit, I hit that as well, I altered my script to basically only test the state files that change in the PR or that push to the repository.