FCREPO-1777 Provide Quality Assurance for fcrepo4-vagrant

[jrgriffiniii]

I have attempted to implement an acceptance testing suite using the beaker framework, within which I am leveraging server-spec in order to validate the state of the virtualized environment provisioned using the BASH scripts.

Unfortunately, as this requires that a base box be downloaded for the provisioning process, it is highly unlikely that this could be integrated with Travis CI without incurring a timeout. Further, attempting to implement module testing for the BASH scripts themselves by using a testing framework such as the Bats would have required that I (significantly) restructure the scripts themselves.

It may be that the approach which I have undertaken is undesirable, that there are superior frameworks or design patterns, or that more modular testing can be implemented. Please freely override or correct my oversights if this is the case.

[ruebot]

@jrgriffiniii for posterity, can you try and get TravisCI set-up with this on your account, so we know for sure whether or not it is possible?

[jrgriffiniii]

@ruebot after many attempts, it appears that I am encountering problems similar to those experienced by Omer Katz:

http://omerkatz.com/blog/2013/6/15/travis-ci-does-not-work-when-you-package-vagrant-boxes-as-a-build-step-yet

Essentially, the kernel release within the Travis CI build environment (2.6.32, appearing to be based off of a release of Ubuntu 10.04 LTS [Lucid Lynx]) does not permit one to successfully install and run any releases of VirtualBox.

I can continue to work to address this, but I shall first need to contact technical support at Travis CI in order to confirm that this is the case.

[ruebot]

Are we certain of the Ubuntu version? I know the PHP TravisCI box is Ubuntu 12.04. It might be worth investigating the box that is being built. I've going down this rabbit hole with some Islandora work :smile:

...but at the same time, it is a bit of a rabbit hole testing a virtualized box inside a virtualized box. Maybe worth a tech call discussion to see what we want to do as a group?

@awoods, @whikloj thoughts?

[jrgriffiniii]

@ruebot The problem lies not with the version of Ubuntu, but with the kernel which is integration with the Linux distribution which they are running within the Travis CI environment:

[...]
The current kernel version is 2.6.32-042stab108.2
Problems were found which would prevent VirtualBox from installing.
Please correct these problems and try again.
[...]

I had assumed this to mean that I could assume a release of 2.6.32 (last supported by 10.04 LTS), but as indicated in the blog post by Omer Katz, this may not be the case.

Further, as you've indicated, technical support at Travis CI has directed me away from attempting to undertake this approach due to concerns in relation to nested virtualization problems:

Hi James,

Thanks for getting in touch.

We do not officially support nested virtualizations like VirtualBox, KVM, Xen on any of our infrastructures. I heard VirtualBox 32-bit has been installed on our Trusty/KVM based beta infrastructure, although I don't know if that ever proved stable.

If you'd like to try and have more questions about our Trusty/KVM infrastructure (http://docs.travis-ci.com/user/trusty-ci-environment) please do let me know.

Have an amazing day!

Hence, I am skeptical as to whether this can be undertaken in a stable and consistent manner between virtualized OpenVZ environments.

[awoods]

@jrgriffiniii, as mentioned here ( https://github.com/fcrepo4-exts/fcrepo4-vagrant/pull/29#discussion_r47448470 ), ideally this script would be completely self-contained and not require any additional requirements on the host system. Is it not possible to have the appropriate utilities installed within the Vagrant box itself?

Here is the error I get when running source spec/accept.sh for the host machine. https://gist.github.com/awoods/ce8e919323b905076557

Instead of debugging this, it would be great if the Vagrant box included everything that would be required to perform this verification from an ssh shell in the box. Thoughts?

[jrgriffiniii]

@awoods:

Instead of debugging this, it would be great if the Vagrant box included everything that would be required to perform this verification from an ssh shell in the box.

To the best of my understanding, the Vagrant Box serves as a system under test (SUT). Hence, following the approach outlined by Joe Ferris' article to which the previous link resolves, I'm fairly certain that the tests must always be run from the test harness (Beaker), lying external to the SUT.

I shall contact members of the Beaker community in order to confirm that this is the case, and that my understanding is accurate. Once I have a response, I can then proceed with diagnosing and resolving the errors identified in your comment.