ci(platform): add `arch-base-latest` & split suites across Travis instances

myii commented 4 years ago

Semi-automated using https://github.com/myii/ssf-formula/pull/40 and https://github.com/myii/ssf-formula/pull/45

myii commented 4 years ago

@dafyddj Another issue that could be covered at the same time:

Many of our formulas have multiple tests suites but we tend to use only run one suite per instance in Travis.
Now that we have seven instances available, can we split testing the three suites across them? I.e.:
- Two of the instances testing dev_server.
- Two of the instances testing install_binary.
- Two of the instances testing prod_server.
- The remaining instance could be kept the same, testing all three suites (and running that one first, so that it doesn't delay the overall run).

myii commented 4 years ago

@dafyddj So I've prepared a comparison, before and after splitting the suites across the various instances:

https://travis-ci.org/myii/vault-formula/builds/589591673 (before):
- Ran for 15 min 27 sec
- Total time 42 min 11 sec
https://travis-ci.org/myii/vault-formula/builds/589598193 (after):
- Ran for 9 min 5 sec
- Total time 23 min 2 sec

The differences are significant. If you're OK with it, I'll add that to this PR.

myii commented 4 years ago

@dafyddj I've added the commit to split the suites in Travis, so this is ready for review.

myii commented 4 years ago

@dafyddj There's bad news on the horizon. As I thought I was done with this PR, all of the instances started failing (when they were working fine in my fork yesterday). Dug down and found out that the latest inspec gem that has just been released (v4.17.7) is resulting in undefined method errors. Added a commit to restrict it to the previous version (v4.16.0) and everything is working again.

A summary of the errors avoided (for the time being):

https://travis-ci.com/saltstack-formulas/vault-formula/jobs/239671364#L2219-L2220 (prod_server):

>>>>>>     Failed to complete #verify action: [undefined method `file' for #<#<Class:0x000055dca642a3c8>:0x000055dca642a2d8>
Did you mean?  fail] on prod-server-ubuntu-1804-2019-2-py3

https://travis-ci.com/saltstack-formulas/vault-formula/jobs/239671365#L1925-L1926 (dev_server):

>>>>>>     Failed to complete #verify action: [undefined method `file' for #<#<Class:0x000055cdc9bcb5f0>:0x000055cdc9bd26e8>
Did you mean?  fail] on dev-server-fedora-30-2019-2-py3

https://travis-ci.com/saltstack-formulas/vault-formula/jobs/239671366#L1515-L1520 (install_binary):

     ×  should not be installed
     undefined method `sysv_service' for #<#<Class:0x0000558e533124a0>:0x0000558e533340a0>
     ×  should not be enabled
     undefined method `sysv_service' for #<#<Class:0x0000558e533124a0>:0x0000558e533340a0>
     ×  should not be running
     undefined method `sysv_service' for #<#<Class:0x0000558e533124a0>:0x0000558e533340a0>

myii commented 4 years ago

@dafyddj Thanks for the review, I've committed the suggestions as a batch. Now due to these repo's settings (as outlined in #45), we can't merge this because of an org-wide issue we're currently having with opensuse. I'll probably fix that in a separate PR, rebase this PR on top and merge it once it's successful.

saltstack-formulas-travis commented 4 years ago

:tada: This PR is included in version 1.2.1 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

saltstack-formulas / vault-formula

ci(platform): add `arch-base-latest` & split suites across Travis instances #43