search

DiSSCo / ecoi_infrastructure_deployer

Vagrant+Ansible project responsible of provisioning VMs for the ECOI services and installing and configuring its subcomponents (cordra nsidr, cordra prov, elasticsearch, mongodb, monitoring tools, etc)
0 stars 1 forks source link

Develop an automated test strategy for the nsidr.org setup #7

Open jgrieb opened 3 years ago

jgrieb commented 3 years ago

We need a set of automated tests that are run every time when the test server is created/ upgraded. However, due to the complexity of the application there are a variety of things that can be tested:

  1. Check that all applications are up and running on the server-side - this can be done with an ansible playbook (see https://docs.ansible.com/ansible/latest/reference_appendices/test_strategies.html )

  2. Test that the services by the cordra-nsidr are provided correctly, for example CRUD operations via http REST API - this could be performed with any scripting library or also with ansible. How could test be run on the DOIP service at port 9000?

  3. Run a test suite to check the functionality of the Web UI with tools like Selenium or puppeteer

jgrieb commented 3 years ago

Point 1 is partly integrated in the ansible deployer setup - if there is a problem during the setup of any of the applications ansible will show an error after the playbook has finished. So there is not so much to do here

Point 3 would be a To-Do for later on in my opinion, but right now the focus is on the functionality and not the UI, so the starting point should be to create tests that run against the API (http + doip) as in Point 2 and check if a DS can be created successfully, check the interaction with the provenance server, etc. @hardistyar what do you think?

jgrieb commented 3 years ago

I developed a first approach of fully ansible-based integration tests. They are found in this ansible script and are executed everytime when the ansible setup is completed.

The tests basically check that CRUD operation on an example digital specimen work as expected and that the provenance records are created accordingly. @hardistyar is this what you were thinking off originally? Right now the tests are using cordra's HTTP REST interface, should we also implement tests against the DOIP or DOIP API for HTTP clients?

hardistyar commented 3 years ago

This is a good first step. Some questions/remarks:

  1. Can the script be executed standalone i.e., any time we want to check the system is working?
  2. Following from (1) can the script be executed against the live system? This could give the basis for extending the suite of possible tests.
  3. It would be good to implement DOIP tests too. Then we have a basis for extending to more sophisticated DOIP tests if needed.
  4. BUT is it a good strategy to invoke this kind of intrusive testing on a live system - might be ok at initial deployment where everything passes and the cleanup is ok before live data is loaded but what about later? If a test should fail and the cleanup doesn't or can't get done properly how does the system recover?

Lastly, I presume this ansible script (and the others) can be used in conjunction with a CI tool like Jenkins or Travis?

jgrieb commented 3 years ago

Thanks for your remarks Alex.

To 1.: Yes, you can run the test script standalone if you limit the "ansible_tags" value to "test_suite" in the config file and than run ansible.

To 2 and 4: I didn't test it yet but with the live system config this should work already. Maybe we should classify the tests into non-intrusive ones (basically only GET-requests & similar) and intrusive ones (post, put, delete). The non-intrusive ones could be performed on the live system at any time, while the intrusive ones should only be performed on the test system (or on the live system after a fresh re-installation of the setup before restoring the data from the backup).

To 3.: Implementing DOIP tests like the REST interface is tested currently directly in ansible will probably get fuzzy as there is no ansible module for it. I played around with shifting the actual tests from ansible into a separate Java project. Tests are then be performed with JUnit, REST-assured and DOIP can be tested via the DOIP java library. Ansible would then only be responsible for invoking the Java program which performs the tests. I think this is probably the way to go now...

To CI/ CD: Jenkins or Travis (and we should also consider github actions) would be responsible for starting the vagrant file, which manages the VMs and invokes the ansible script to be executed on one of the VMs. We should give some further thinking into how exactly we are planning to incorporate CI/CD. But starting a Vagrant job with github actions everytime a commit is made seems possible. The questions remains whether this Vagrant script runs the "local setup", i.e. VMs are created within github actions (there might be processing power limitations to running 6 VMs at the same time) - otherwise the github action would be responsible for running the test environment on AWS.

jgrieb commented 3 years ago

As mentioned above, I changed the testing strategy. I implemented a standalone Java project that can be run to test the HTTP and DOIP interface of the nsidr.org setup. The library is currently found here: https://github.com/jgrieb/nsidr-api-tests A compiled JAR file of the library is uploaded to S3. Then, I updated the Vagrantfile in the new branch. After every provisioning with the Vagrant script, the Java-based tests will be executed automatically. The required binary and configuration files are created dynamically by the script in a folder 'test_suite' (is ignored via .gitignore). These updates are currently in a new branch: https://github.com/DiSSCo/ecoi_infrastructure_deployer/tree/run_tests_from_vagrant_script