search

saltstack-formulas / vault-formula

http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html
Other
15 stars 59 forks source link

ci(travis): use `kitchen verify` instead of `kitchen test` #42

Closed myii closed 4 years ago

myii commented 4 years ago
myii commented 4 years ago

@dafyddj Remember our conversation about this a few months back? To welcome you to the organisation, a simple PR for you to review first up. Feel free to merge (or not), as you see fit.

aboe76 commented 4 years ago

@dafyddj almost merged this one....but after reading @myii comments, welcome to the organisation!!!

dafyddj commented 4 years ago

The conversation in Slack was as follows:

<dafydd> @myii on the subject of `test` vs `verify` I'm not sure I was getting my point across yesterday.I ran a test, the results of which you can see here: <travis-ci.org/dafyddj/vault-formula/jobs/556966438#L3935>After a test run with `verify` each test suite leaves a docker instance running taking up a certain amount of memory (here 5%)so running around 20 test suites will eventually run out of memory on the Travis instance.Admitted
18:22onslack
no formula will ever test 20 suites - so if the consensus is to use `verify`, I will change it - but using `test`, which is such a simple change, no memory is "leaked" after a test suite
18:24onslack
<myii> In terms of number of test suites, we're restricting to 6 maximum, due to concurrency limits. See <saltstack-formulas/template-formula #118> for more info. Essentially, we're not going to hit that limit any time soon.
18:25onslack
<myii> Out of interest, where did you find the Travis memory usage?
18:26onslack
<myii> Never mind, seen it in your commit.
18:29onslack
<dafydd> I don't want to labour the point, but test suites are different to Travis concurrent jobs
18:29onslack
<dafydd> or platforms
18:29onslack
<myii> I seriously understand your point.
18:29onslack
<myii> I'm moving around it to the problem we're facing as an organisation.
18:30onslack
<myii> When we first started implementing pre-salted images and `semantic-release`, we were reaching 1 hour backlogs across all of the formulas.
18:31onslack
<myii> So we reduced the number of jobs per formula; but we also want each job to finish as quickly as possible (hence, the pre-salted images).
18:32onslack
<myii> There may be little time saved by the extra implicit `kitchen destroy` but every bit helps, especially if it makes no difference to our testing itself.
18:32onslack
<myii> We just want these jobs to finish ASAP.
18:34onslack
<dafydd> ok, I get it, I've made my point - I'll put in a PR to switch to `verify` for consistency

For future reference, docker resource usage in Travis can be assessed by adding the following to .travis.yml.

script:
  - bundle exec kitchen verify ${INSTANCE}
  - docker stats --no-stream
  - free -h
dafyddj commented 4 years ago

@myii @aboe76 Thank you for the welcome! And thanks for the easy first review! I hope I can make a useful contribution to the organisation.

myii commented 4 years ago

Thanks for the merge, @dafyddj. I hope it didn't take too much out of you!

saltstack-formulas-travis commented 4 years ago

:tada: This PR is included in version 1.2.1 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: