search

google-code-export / mb-unit

Automatically exported from code.google.com/p/mb-unit
0 stars 0 forks source link

Add first-class support for profiles. #108

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Goals:

1. We want to use one set of data for running tests in the testing
environment and a different set in the staging and production environments.

2. We want to forcibly disable execution of certain tests in certain
environments where they might be unsafe to run or where they might lack
necessary resources (such as in production).

In an in-house test suite used by functional testers, we found it very
helpful to define a notion like a profile in MbUnit v2.  We introduced a
hack that would prompt the user to pick a profile the first time the tests
were run and then remember the selection until the test runner was shut down.

This worked well enough but it was clunky.

So instead we should introduce a notion of a "profile" (or some other
conditional behavior).

1. Extend the data source attributes with a Profile property.

[Row(1, 2, 3, Profile="Test")]
[Row(4, 5, 6, Profile="Stage")]
[Row(7, 8, 9, Profile="Test,Stage")]
[Row(10, 11, 12)] // the default would work in any profile

2. Extend the test attributes with a Profile property.

[Test(Profile="Test")]
// etc...

Now the question becomes how to enumerate the available set of profiles. 
Obviously we can do that by tracking all profiles seen (and we should do
that).  However we might also wish to provide extended descriptions for
profiles.

[assembly: Profile("Test", "Runs tests against the testing environment.")]
[assembly: Profile("Live", "Runs tests against the production environment.")]

Profiles can then be exposed to the test runner (by way of extra properties
on the TestModel) which can then prompt the user as needed.

Another option for declaring profiles would be to put them in a
configuration file associated with the test assembly.  However that does
not significantly change the rest of the story.  It also seems a bit clunky.

Comments welcome.

Original issue reported on code.google.com by jeff.br...@gmail.com on 10 Dec 2007 at 9:54

GoogleCodeExporter commented 9 years ago

Original comment by jeff.br...@gmail.com on 6 Jan 2008 at 7:07

GoogleCodeExporter commented 9 years ago

Original comment by jeff.br...@gmail.com on 6 Apr 2008 at 2:22

GoogleCodeExporter commented 9 years ago 
Another use for profiles is be to leverage XML processing instructions in the 
test
assembly configuration file.

eg.

<?if TEST?>
  <config for test environment>
<?endif?>

Original comment by jeff.br...@gmail.com on 11 Oct 2008 at 6:09

GoogleCodeExporter commented 9 years ago

Original comment by jeff.br...@gmail.com on 19 Nov 2008 at 11:58

GoogleCodeExporter commented 9 years ago 
Could enable additional properties to be passed in besides profiles.  Perhaps 
let
individual test frameworks decide how they want to consume these options.  So 
MbUnit
might consume a Profile property (and possibly other options as well) but a 
different
framework might use other things.

One neat trick would be to allow test frameworks to expose the set of 
properties they
can use, possibly based on the results of initial test exploration.  In 
particular,
it would be neat if they could publish well-known values for those properties.

So for integration testing someone might use an "Environment" property to 
specify the
target environment.  Its allowed values might be "Dev", "Test", "Stage", 
"Prod".  The
property and its values would be declared using a config file or 
framework-specific
attributes.

This mechanism might also enable a data-driven test to be executed with specific
values bound to its parameters by the user, given appropriate framework support.

Original comment by jeff.br...@gmail.com on 20 Mar 2009 at 10:02

GoogleCodeExporter commented 9 years ago

Original comment by jeff.br...@gmail.com on 23 May 2009 at 1:52

GoogleCodeExporter commented 9 years ago

Original comment by jeff.br...@gmail.com on 29 Jun 2009 at 8:33

GoogleCodeExporter commented 9 years ago

Original comment by Yann.Tre...@gmail.com on 18 Jul 2010 at 8:09

GoogleCodeExporter commented 9 years ago

Original comment by Yann.Tre...@gmail.com on 14 Jun 2011 at 5:53