search

saltstack-formulas / openvpn-formula

Setup and configure openvpn server and client.
http://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html
Other
35 stars 114 forks source link

Windows testing using kitchen-vagrant #112

Closed dafyddj closed 5 years ago

dafyddj commented 5 years ago

I'm putting this out there to gather some opinions. Would this be useful to add to the formula, despite the fact that the tests can't be run in any CI/CD service that I know of?

Testing requires Virtualbox and Vagrant on the local machine. The OpenVPN server is first run on a linux box, and the tests are subsequently run on the Windows box.

KITCHEN_YAML=kitchen.windows.yml kitchen converge default-ubuntu
KITCHEN_YAML=kitchen.windows.yml kitchen verify default-windows
myii commented 5 years ago

@dafyddj Apologies for the poor-quality communication earlier, both here and in Slack -- I was out and about so it got a bit messy. I'm glad you figured out what I was trying to say!

I'm putting this out there to gather some opinions. Would this be useful to add to the formula, despite the fact that the tests can't be run in any CI/CD service that I know of?

I'd very much welcome Windows testing, even if it was only local. It would help with PRs that have got stuck, such as the following:

In terms of CI/CD, there's a potential solution. You may have noticed that Cirrus CI has been enabled for this org's repos and is being used for a few of them (e.g. rkhunter and syslog-ng). They also have Windows Containers available, which could be useful. We've had contact with Cirrus CI and they were responsive and helpful, both on GitHub and via. e-mail, such as the discussion going forward from here:

dafyddj commented 5 years ago

I have had a little play with kitchen-docker using Windows containers (support was only added a matter of weeks ago), and there's a slight gotcha in that files cannot be uploaded to the container while it's running - so, at least at the moment, it's not really a viable testing method.

dafyddj commented 5 years ago

@myii care to take another look at this?

I've changed the approach a little: pre-Salted Windows image; re-use of Inspec tests and kitchen.yml. Comments welcome.

myii commented 5 years ago

@dafyddj Sounds good! I'll try to have a look at this soon.

myii commented 5 years ago

@dafyddj Whatever decision is reached, it would be good to update the testing section in the README to explain how to run the local Vagrant-based testing.

dafyddj commented 5 years ago

While I look over this, what do you think of this approach (1037162eb1aac2be54f39d55d454a9d8b48f0d2c) whereby .sls files are rendered to .yaml before being linted by yamllint? (kitchen.yml could be rendered in a similar way.)

dafyddj commented 5 years ago

Resulting Travis output here.

dafyddj commented 5 years ago

@dafyddj Whatever decision is reached, it would be good to update the testing section in the README to explain how to run the local Vagrant-based testing.

Yes, this is on my TODO list.

myii commented 5 years ago

While I look over this, what do you think of this approach (1037162) whereby .sls files are rendered to .yaml before being linted by yamllint? (kitchen.yml could be rendered in a similar way.)

@dafyddj This has crossed my mind plenty of times, since getting my hands dirty with all of the linters. It's nice to see a working method. It definitely appeals to the perfectionist in me but that's not necessarily a good thing! Having cast my eye over a lot of debug output, it's obvious that we'd have to use a fairly relaxed set of yamllint rules, which would be easy to do via. a secondary configuration file (that can be customised per formula).

So what are my concerns? They're a bit elusive right now but I suppose one is that we could end up distracting ourselves unnecessarily, or giving ourselves too many hoops to jump through. Another is that the lint won't cover the whole of each source file; the rendered files will miss sections due to conditionals, etc. The issue surrounding jinja_env:trim_blocks also springs to mind -- in that, we could end up with excellent rendered output, which actually breaks when used by consumers.

I'd still like to see this in action, though. How about we do this in a separate job with jobs:allow_failures set, like we had with rubocop? That will give us a chance to see how it works out. We can also share our findings with the others, to attempt to achieve a consensus.

Please don't mistake this as a negative response in any way. I would definitely use this in a personal capacity, within a well-defined environment. My feedback here is more measured, trying to consider the wider implications.

myii commented 5 years ago

@dafyddj Another reason for having a general purpose kitchen.vagrant.yml: supporting FreeBSD images as well. There's a fairly long thread on Slack but today's update is here:

- - -
16:06 myii @​kp We're getting closer to establishing Vagrant-based testing again: https://saltstack-formulas/openvpn-formula #112. We could easily add the FreeBSD images to the kitchen.vagrant.yml file. How are you getting along with the pre-salted images? CC: @​noel.mcloughlin.
16:10 kp @​myii thanks for the pointer, was pretty busy with $real_job issues, I hope I can prepare something meaningful this weekend
16:10 myii Excellent, look forward to it.
myii commented 5 years ago

@dafyddj Is this still a WIP?

dafyddj commented 5 years ago

Yes, still have a couple of things to add to it.

myii commented 5 years ago

@dafyddj OK, please ping me when you're ready for the next stage.

dafyddj commented 5 years ago

@myii I've added to the documentation and updated my Vagrant box (to version 2019.10.141), so this PR is ready for further review.

saltstack-formulas-travis commented 5 years ago

:tada: This PR is included in version 0.15.4 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: