Appliance testing framework

[JedMeister]

It'd be great if we had an appliance testing framework. This could be used to create a raft of test we could do on appliances.

@qrntz started wortk on a framework that leverages PhantonJS.

@Dude4Linux suggested that perhaps Selenium might be a better option. @qrntz had some concerns about that having a lot of x11 dependencies but it seems that the Selenium Server is desinged to run the tests headlessly (see here). Actually there is a 3rd party PhantonJS "driver" avaialble for Selenium (called GhostDriver).

Here are some other links that may be of interest: http://docs.seleniumhq.org/download/ https://github.com/angular/protractor/blob/master/docs/server-setup.md https://www.npmjs.com/package/selenium-standalone http://nightwatchjs.org/

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[SiKing]

Just noticed this today. Selenium is a browser-app automation framework, so you would only be able to automate browser stuff. Not sure if that is what you are after. GhostDriver gives you Java bindings for PhantomJS. This is good, assuming you want the framework written in Java. Also worth considering, from the GhostDriver page: "I have been away from this project for 2+ years. ..." Anyway, if this is the direction you would like to go, I can definitely help out.

[JedMeister]

TBH @SiKing I have no idea about the intricate details of this stuff. It was just a dump of some info that I had floating around.

Bottom line is that we really need to implement some automated testing. I think that there are plenty of ideas and options around for the non-browser side of things; but it would be nice to double check things like the admin password (set by inithooks) allows login and stuff like that. The downside of that is that we will need to write a lot of tests which won't be that exciting... And we will probably still want to double check the screenshots (that the testing takes as it goes).

We probably really need to develop some clear goals, then build a plan of how we might attack this. Then we can zoom in closer on the details. Like I said, Anton has already done a fair bit of work on a PhantonJS based framework but it's been low on the priority list so keeps getting pushed back...

Regardless, thanks for your interest. Perhaps we need to start a new page on the wiki so we can develop the ideas and clarify the goals...?!

[a3s7p]

It's CasperJS, actually, which offers a superset of PhantomJS features (so more high-level, less boilerplate). Here it is.

To start with, it would probably be useful for me to lay out my vision of how this should work in practice somewhere, since I've written the bulk of code relating to this so far and faced quite a few practical issues along the way. A wiki page would do. Afterwards everyone can contribute.

[SiKing]

I work as an SDET. So testing getting a "lower priority" I am used to. :) The project I was just on had their budget cut, so they decided to save money by dropping testing all together.

[Dude4Linux]

It might help to discuss the various use cases for a testing framework. I think @JedMeister is referring to a framework for testing appliances during or after the build process which would help him automate the process of creating a new release. Previously we've made use of the testssl.sh script to help improve appliance security. Recently, I suggested that Lynis was another tool that would be useful in ensuring that appliances were following security best-practices. My first introduction to DevOps was at the 2013 DrupalCorn conference. Yes that's right, DrupalCorn. Here in Iowa, the three major Universities have all adapted Drupal as the CMS of choice. They take turns hosting a summer Drupal Camp. In 2013, I was investigating the idea of bootstrapping a small business which would serve Churches and other non-profits that had needs that were not being met in the marketplace. Having previously worked for a small business doing IT support, I felt that using DevOps and automation was the key to building a successful business. Two of the takeaways that I got from that 2013 conference were Ansible and Selenium. That was the impetus for starting work on the Ansible appliance. For some months now, I've been using the Ansible appliance in conjunction with the LXC appliance to run simple tests on the 'proxmox' builds that @JedMeister provides. The great thing about Ansible is that it can easily run external scans such as testssl.sh or security auditing as in Lynis. I did find that Ansible's built-in html test capability was rather limited so that another appliance based on CasperJS, Selenium 2.0, or possibly both makes a lot of sense. I look forward to your proposal @qrntz.

Opps forgot to include the links: testssl.sh https://testssl.sh/ Lynis https://cisofy.com/lynis/ DrupalCorn http://2016.drupalcorn.org/

[JedMeister]

@qrntz - Thanks for the clarification and awesome to have a quick look over your work to date. :smile: I agree that it may be good to share your vision/plan for this. Perhaps after you're done with the other priority stuff we're working on currently, you could briefly do that? Then even though we have higher priority stuff to be working on, perhaps there is room for others to keep working on this while we're busy doing other things? Just a thought...!?

@SiKing - Thanks again for your interest and offer of assistance. :+1:

@Dude4Linux - Thanks for adding some clarification to the mix! You are correct, I am talking about some web UI testing stuff. The plan is for it to also take screenshots as it goes. This has a dual purpose. It will allow us to view any potential issues, as well as give us current screenshots so we can keep our appliance library screenshots on the website up to date and relevant. Actually it might be useful for you to share your work and the testing framework that you have already developed?! Perhaps we could/should integrate your work and Anton's into an additional appliance based on TKLDev?! I.e. add the LXC functionality, the Ansible functionality and the testing scripts etc? Perhaps that's overkill at this point, but it would be pretty damn cool to be able to build an appliance in the "proixmox" format, run it in LXC and then run a battery of tests on it! Also if you have already publicly shared your work and I either missed it or forgot about it, please accept my humble apologies. Finally as a total aside, for some reason I thought you were in the UK (Bristol springs to mind for some reason??) but Iowa is obviously in the US!?!

[Dude4Linux]

@JedMeister My vision for a DevOps suite of appliances is to keep blocks of functionality in separate appliances and provide API's that link them together kind of like Lego building blocks. The functional blocks can then be combined in various ways to suit the needs of the end users. A lot of cloud development, OpenStack, Docker, etc seems to be moving in this direction. I had to add some code to both the Ansible and LXC appliances to allow them to work together. In the future, I hope to do the same with the Jenkins and GitLab appliances. Using this model would mean that if someone prefers Salt, Puppet, or Chef over Ansible, they could develop an appliance that meets their needs and still take advantage of existing appliances. Someone who prefers Docker to LXC for container management could develop a Docker appliance with similar functionality. Currently TurnKey lacks the developer bandwidth to tackle all of the possible combinations, but I think you have done a great job in creating an open and welcoming environment for new developers.

Regarding TKLDev, I believe that the ability to build any of the supported formats including "proxmox" should be integrated into a future version of TKLDev. I think this is a natural extension of the "build" function. I agree that any built-in tests that run in the fab-chroot environment should be included there. Possibly a Lynis security audit belongs here as it is just a shell script that does not need to be installed. I took a look at Anton's @qrntz work and it looks to me like it could be adapted to capture screenshots during the build sequence. My feeling is that including LXC and Ansible in TKLDev would be overkill and harder for new developers to master. I do recommend that we look into including jinja2 templating into the next version of TKLDev and also adding back the ability to install patches during the build sequence. Several times I've had to resort to an overlay when I really wanted to just patch an installed file. Along that same vein, I'd like to see more of the pre and post script options from TKLPatch. Anton and I had a hell of a time getting some of the security changes applied at the right time for the 14.0 release. I know someone else is working on the next version of TKLDev, but I'd like to volunteer to help as well.

Regarding Ansible, I have shared some of my work on plays and playbooks in Gists and also included some in the 14.2 branch. https://github.com/Dude4Linux/ansible/tree/updates-for-v14.2/overlay/usr/share/ansible/containers I'm not particularly proud of any of them. Like most Ansible n00bs, I started putting everything into large playbooks that run like scripts. Now I'm faced with breaking them up and creating equivalent Ansible roles. My biggest stumbling block has been that the term 'role' carries meaning and some assumptions in my mind that I'm beginning to realize are not true in the way Ansible uses the term. I hope to get my head around it before it's time to release 14.2. As for the UK, you might have confused me with Jon Hawkesworth who's helping me add Windows support to the Ansible appliance. He's based in London (not sure about Bristol).

[JedMeister]

Let's make providing some level of automated testing to the v14.2 build part of the plan. Exactly what it will do and what level of testing it provides can be defined/clarified/decided at a later date.

Here's something that I think looks really cool: http://os-autoinst.github.io/openQA/

@qrntz - assigning you to this issue for now as I'd really like you to publicly state your vision (IMO without that this issue is somewhat blocked).

@Dude4Linux:

@JedMeister My vision for a DevOps suite of appliances is to keep blocks of functionality in separate appliances and provide API's that link them together kind of like Lego building blocks. The functional blocks can then be combined in various ways to suit the needs of the end users. A lot of cloud development, OpenStack, Docker, etc seems to be moving in this direction. I had to add some code to both the Ansible and LXC appliances to allow them to work together. In the future, I hope to do the same with the Jenkins and GitLab appliances. Using this model would mean that if someone prefers Salt, Puppet, or Chef over Ansible, they could develop an appliance that meets their needs and still take advantage of existing appliances. Someone who prefers Docker to LXC for container management could develop a Docker appliance with similar functionality.

Great vision/plan! I :heart: it! :+1:

Currently TurnKey lacks the developer bandwidth to tackle all of the possible combinations, but I think you have done a great job in creating an open and welcoming environment for new developers.

Thanks for the kind feedback. And you are definitely right about our current (shortfall of) "developer bandwidth" WRT to exploring all the possibilities. However if we have something that has been done and works (as per your vision) but with the flexibility of it to be used as sort of a "template" which could be used with alternate possibilities then that's be extremely cool.

Regarding TKLDev, I believe that the ability to build any of the supported formats including "proxmox" should be integrated into a future version of TKLDev. I think this is a natural extension of the "build" function. I agree that any built-in tests that run in the fab-chroot environment should be included there.

I agree. FWIW in case you weren't aware, it is already publicly available but isn't pre-integrated and is not particularly well documented. Have a look at buildtasks. One major shortcoming is that it (or more correctly; the OVA build) currently relies on the non-open source VMware OVFTool...

Possibly a Lynis security audit belongs here as it is just a shell script that does not need to be installed. I took a look at Anton's @qrntz work and it looks to me like it could be adapted to capture screenshots during the build sequence.

Hmm interesting idea. Definitely has merit.

My feeling is that including LXC and Ansible in TKLDev would be overkill and harder for new developers to master. I do recommend that we look into including jinja2 templating into the next version of TKLDev and also adding back the ability to install patches during the build sequence. Several times I've had to resort to an overlay when I really wanted to just patch an installed file.

So something like https://github.com/turnkeylinux/buildtasks/pull/11 ?

Along that same vein, I'd like to see more of the pre and post script options from TKLPatch. Anton and I had a hell of a time getting some of the security changes applied at the right time for the 14.0 release. I know someone else is working on the next version of TKLDev, but I'd like to volunteer to help as well.

The more the merrier. Although I think it would be really good for us to try to communicate what/when we are going to work on different things. Perhaps we could even go as far as to develop some sort of read map for TKLDev? Otherwise if we're not careful we might end up treading on each others toes! :smile:

Regarding Ansible, I have shared some of my work on plays and playbooks in Gists and also included some in the 14.2 branch. https://github.com/Dude4Linux/ansible/tree/updates-for-v14.2/overlay/usr/share/ansible/containers

Great. Sorry I missed that. TBH I haven't even played with Ansible so really need to spend some time on it. One thing I do really like though is that it's agent-less...

I'm not particularly proud of any of them. Like most Ansible n00bs, I started putting everything into large playbooks that run like scripts. Now I'm faced with breaking them up and creating equivalent Ansible roles. My biggest stumbling block has been that the term 'role' carries meaning and some assumptions in my mind that I'm beginning to realize are not true in the way Ansible uses the term. I hope to get my head around it before it's time to release 14.2.

Thanks for sharing your insight. I haven't barely even looked at Ansible yet so you've probably helped me avoid a pitfall! Thanks so much for sharing your vision and ideas. It get's me a bit excited really! :+1:

As for the UK, you might have confused me with Jon Hawkesworth who's helping me add Windows support to the Ansible appliance. He's based in London (not sure about Bristol).

I obviously got you confused with someone; but I'm almost certain it wasn't him, it was from a long time ago. It's so weird I can still see "Bristol, UK" next to your avatar in my mind's eye... Obviously got my wires crossed somewhere. TBH I wasn't even sure where Bristol was until I googled it! Oh well... Might need to get some sleep!

[Dude4Linux]

@JedMeister I took a stab at implementing one of my suggestions above, namely being able to apply patches during the build. I thought I had been able to do that with TKLpatch but my memory was faulty. I do remember having the need to apply an upstream patch to fix a problem where the patch had not been backported. This was my first attempt to explore the TurnKey fab and modifying make files. I'll open a tracker issue and a pull request after I've had a chance to do some more testing. You can checkout the changes here https://github.com/Dude4Linux/fab/tree/add-fab-apply-patch

[JedMeister]

FWIW there is a pending PR from @jstruebel where he has already implemented more-or-less what you have suggested. Apologies that I didn't note it previously. His tack was slightly different as it leverages buildtasks instead. See: https://github.com/turnkeylinux/buildtasks/pull/11

The significant difference is that his implementation is a new buildtasks script so probably better supports batch processing and/or processing existing ISOs; whereas yours does it during the initial build process so would support easier testing at root.patched.

Having said that; if you wanted to test a patch within root.patched you could already do it pretty easily with tklpatch, like this:

make root.patched
tklpatch-apply build/root.patched example-patch.tar.gz

or to using your directory tree style:

make root.patched
tklpatch-apply build/root.patched patches.d/

BTW there are also other pre-existing TurnKey buildtasks scripts which do similar stuff. E.g. bt-maintenance which is what we use internally for basic maintenance releases (it leverages tklpatch to support batch patching of existing ISOs). It was updated for the v14.1 release, although wasn't used in the end (except for Canvas; as we decided to rebuild the rest from scratch).

As an aside (but sort of relevant here); Alon has been working on an idea to break up components of builds into "build units" i.e. break it up into obvious blocks. He's labelled it unit.d. I'm really hoping that it will make moving stuff into (and potentially out of) common much nicer. It will hopefully make re-using code from other appliances really easy too (i.e. it may still need to be tweaked to the local specific requirements but rather than copying code, you could copy whole files then just apply relevant minor tweaks). It should also make code consistency much easier to achieve. Anyway, I haven't actually seen what he has done yet so can't really elaborate more on that but he'll be sharing soon (if he hasn't already and I just missed it).

As another aside (which is very relevant here); we are looking to set up a better development communication channel. We have the forums and we have github; but we've realised that we really need a better way for TurnKey developers to communicate between themselves. So we are looking at setting up a slack channel to test it out (and an opportunity to see what all the slack hype is about). We'll also probably have a play with mattermost (slack-like open source tool) too and see if that will be a better/equivalent tool. Obviously we'd prefer to go with the open source option if it suits our needs...

[Dude4Linux]

I wasn't aware of Jonathon's @jstruebel work on patches in buildtasks, but it is good to know there are others who see the need. I can't remember if @alonswartz added tklpatch-apply to version 0.93 at my request, but I do know he added the pre- and post- config script options in 0.92 at my suggestion.

Having the ability to apply patches during buildtasks is only beneficial to the maintainers. My feeling is that some of the functionality of buildtasks should be moved into TKLDev where it can be used by app developers. Since TKLDev was meant to replace TKLpatch, I think that the missing functionality should be added to TKLDev rather than trying to use TKLpatch instead.

I do have one question. Is it a requirement to apply patches in 'common'. I think Jonathon's work suggests that it is. If so, I'll have to modify my code some more.

I was aware of Alon's work on unit.d. I spotted it when I checked out the latest fab. Keep me posted about the slack channel.

[JedMeister]

Apologies on slow reply. Actually this has got way off topic. I think it was my fault! :blush:

I pretty much agree re buildtasks. It's simple to install, but unless it's included (and documented) many developers wouldn't even think to use it. I think that buildtasks should probably become a default component of TKLDev.. Because I like to use LXC whenever possible, buildtasks is an invaluable part of lots of stuff I do with TKLDev. But I guess I'm not your average Joe either! :smile:

Don't quite understand what you mean by "apply patches in 'common'"?

[Dude4Linux]

@JedMeister Not a problem. I know you've had your hands full getting the 14.1 release out the door. Thanks for the kind mention.

In https://github.com/Dude4Linux/fab/tree/add-fab-apply-patch, I'm trying to recreate the functionality of tklpatch-apply into fab, adding a new command fab-apply-patch. Appliance specific patches are placed into an optional directory 'patches.d' alongside the config scripts in 'conf.d'. The question I had in mind is what to do with patches that apply to all appliances. Say you have a need to apply a patch to webmin. My understanding is that currently patches are applied upstream to the turnkey webmin packages. My first thought was to create another directory in the common area, /turnkey/fab/common/patches, and add a stanza to the makefile that installed any patches found there. Later, it occurred to me that patches could also be added to core, i.e. /turnkey/fab/products/core/patches.d, but I haven't yet tested that option. It's raining today so I have an opportunity to do some more testing.

[jstruebel]

@Dude4Linux Not sure what you're trying to accomplish with patching the appliances, but my use case was trying to collect/automate customizations that I would make to the configurations once I had the appliance running. After thinking through it and discussing it some with @JedMeister, I've come to the conclusion that most of what I was trying to do would be better implemented as ansible plays/playbooks. That would allow them to be easily shared and implemented by any TKL user without needing to setup TKLDev and rebuild the appliance. My thinking is that any general enhancements or fixes to the appliances can just be made directly to the git repo, and things that are specific configurations, e.g. setting up Postfix to forward to an SMTP relay, would be part of a library of ansible plays/playbooks like what you're including for 14.2. I don't see how the patches still fit in that architecture, but if your vision for using patches complements it, I'm interested in hearing about it.

All that said, my setup was to create a patches directory alongside the products directory with the various appliances. If I had any patches that depended on other patches being applied (I was trying to keep each patch specific to a certain config and modular) then those dependencies were included in the patches subdirectory of the patch. My modification of the bt-iso script referenced in the PR above uses tklpatch-apply on the root.patched deck during the build. I created a "super patch" that included all of the patches that I wanted to apply to a certain appliance and then used that when running the bt-iso-patched script.

[Dude4Linux]

@jstruebel, My use case was a bit different than your's. Besides the official appliances I've worked on, I've built several one off appliances that never made it to the official list, for example a Unify controller for managing Ubiquiti WiFi radios. On several occasions, I've wanted or needed to apply an upstream patch that wasn't included in any of the package repos. The solution was always install the patch utility in the appliance and then download and install the patch in a configuration script (conf.d/main). I've always thought this was messy.

I've added the ability to install patches to TKLDev by modifying fab, basically copying the fab-apply-overlay and applying patches instead of copying files. I choose this feature as a learning challenge to understand more about how fab and hence TKLDev function behind the scenes.

More explanation can be found at the end of README.rst.

I agree that if you are building and managing many appliances, setting up an Ansible server and learning YAML is probably the way to go. As I pointed out to @JedMeister, the learning curve for Ansible is pretty steep. I was drawn to it as opposed to Chef, Puppet, or Salt because it was agent-less. I immediately grasped how Ansible worked because I had previously been using remotely executing scripts to apply my personal customizations. For example see turnkey-setup.

Ansible makes extensive use of jinja2 templates for creating and managing configuration files. I've been thinking about ways that jinja2 could be used during the build process and also during firstboot. Confconsole has issues being able to handle all of the configurations it encounters, e.g. the LXC appliance. Would being able to pre-seed a jinja2 smart template(s) during the firstboot have any value for your use case?

[jstruebel]

I see, you were trying to apply patches from upstream against the installed applications not a tkl-patch. Even though I think you could use tkl-patch to do it, your way is simpler by just using the patch file itself rather than wrapping into a tkl-patch. I agree that your use case is better served with the fab modifications.

As far as pre-seeding a jinja2 template, I'm not sure yet. I wasn't familiar with it so I took a quick look at the link you posted. I don't quite understand how it would work with firstboot and Confconsole, but then again I'm not very familiar with their inner workings since I've only copied/modified a few of the firstboot scripts. However, the way I've been using Turnkey Linux is as VMs (either KVM or OpenVZ/LXC) through Proxmox on my personal server. I haven't stumbled across a clean way to use preseeding on a new VM, especially when bringing up a KVM one from an ISO.

Possibly the LXC appliance provides pre-seeding in a seamless manner when creating new containers, but I haven't tried replacing Proxmox with it. The primary thing holding me back so far is that there isn't a web UI for it. I like having all of the necessary options presented in the UI when I'm creating a VM so that I don't have to remember the exact syntax of the command line.

So in my situation I think the Ansible route will work better than the pre-seeding. Now I just need to spend some time with it so I can learn it.

[a3s7p]

Sorry for making you wait. Input welcome.

https://github.com/turnkeylinux/tracker/wiki/Testing-Framework-Overview

[JedMeister]

FWIW, @qrntz relatively recently upgraded his POC code so it works with current versions of PhantonJS/CasperJS.

Since then @ongle has taken over and is madly updating javascript to test the appliances.

Whilst it's primarily intended as a build testing framework, ideally I'd like to see be able to support use on the LXC appliance too. I.e. download the current LXC template and launch that, rather than clone and build from source on TKLDev.

[JedMeister]

Ok so we now have clicksnap! It's still not battle tested or tightly integrated, but it's a thing...

As such, I'm going to close this issue now. If there are any issues/questions re clicksnap, let's do that in a new issue.