search

SatelliteQE / robottelo

Robottelo is a test suite that exercises The Foreman.
GNU General Public License v3.0
61 stars 112 forks source link

Destructive tests #4088

Closed ldjebran closed 5 years ago

ldjebran commented 7 years ago

Destructive tests are tests, that cannot run in parallel with other tests as to achieve their test functionality they have to stop, start, restart the server application, some of them have to destroy and restore the existing data or to achieve an upgrade from version x to version y.

Even if the test will run sequentially and if something goes wrong (in case of test failing or for example restoring the data or bringing the service back online) all the other remaining tests have no chance to even start running.

Classification of destructive tests:

  1. tests that need a service restart
    • UI/CLI/API test_positive_restart_promote_via_dynflow
    • UI/CLI/API test_positive_restart_dynflow_publish
  2. tests that need a service restart + data destroy + data recovery
    • HotBackup test pipeline #3905
  3. tests that need a service restart + service version modification + data modifications

We needed to find a well designed running scenario solution to these tests categories. Waiting you comments, Please do not hesitate to add the tests or category you found it missing.

rochacbruno commented 7 years ago

I asked the same question on #3905 about how to deal with backup tests as it demands stopping the system to perform the restore.

ldjebran commented 7 years ago

@rochacbruno yes I know, but we have a collection of such tests raising and we have to find a general solution for them all, it seems to me running at the end of the pipe line is a very temporary solution.

ldjebran commented 7 years ago

@rochacbruno or we should have two or three sat instances running to be used in case of instance failure

sghai commented 7 years ago

I think these tests can be run once all tiers are finished. There are manly tests that we may need to automate around installer options. All of them can be covered this way.

We can run them after long running tests or after rhai tests and call them as system_tests.

ldjebran commented 7 years ago

@sghai, that way in case of services that restart the system, we can try to test if the service is alive and start it if down, and let backup restore to the end of the queue how we should mark the tests in your case? - this the main question and how about the upgrade tests? - for the moment I have no task for this

sghai commented 7 years ago

@ldjebran : I think we can anytime check the katello-service status or any individual service status before/after running the test. On other thought, this could be part of teardown to revert the service back once test is done, If I understood your question correctly.

Also, about upgrade tests, upgrade is completely being handled via automation-tools

ntkathole commented 5 years ago

Please feel free to re-open, if this still needs fixing.