search

SatelliteQE / robottelo

Robottelo is a test suite that exercises The Foreman.
GNU General Public License v3.0
61 stars 112 forks source link

Proposal: Changing how we think about test cases and endpoints #7358

Open pgagne opened 4 years ago

pgagne commented 4 years ago

This proposal is a WORK IN PROGRESS

Currently in robottelo we have a few high level constructs that basically force us to put Test cases we right into buckets of on of API, CLI, and UI test modules and test cases (ie cli/test_activation_key.py)

This is mainly enforced by a few different things.

  1. Most (all?) test cases are either subclassed from APITestCase or CLITestCase (see https://github.com/SatelliteQE/robottelo/blob/master/robottelo/test.py )

  2. We store test cases in high level directories by endpoint `tests/foreman/{api,cli,ui,etc}

While this may of made sense at one time, I think moving forward we should rethink how we categorize test.

I often find myself coming up with ideas for test cases, then struggling to fit them into these narrow buckets, mostly due to the fact that those test cases transcend endpoints (ie they involve working in both the CLI, API, and maybe even the UI". While we have created multiple "special cases" for these, I think these sorts of test should be the rule, rather then the exception.

I propose that we move to a new system, perhaps categorizing test by high level features or categories. Here is one such way to catagorize them: 

tests/foreman
-

to properly categorize them we could have a directory call tests/foreman/features/ with subdirectories of each feature. Existing test cases can get copied into these test

ntkathole commented 4 years ago

It is not convincing as per benefits Vs efforts/changes required ratio. Can you please share more on benefits of changing directory structure as per above?