Ordered specs

[raghur]

I'm writing outside-in API tests... IN that context, having a known execution order of classes helps. This PR sorts spec types by name so that tests can run like S01_Something_Specs S02_Something_else_specs

[amirrajan]

Can you give me an example of one of your specs? I don't think this is an issue, just want to see what specifically you're dealing with.

[raghur]

As I said, I'm writing API tests generated from Swagger specs which exercise the system end to end. In that sense, data setup is a costly affair time wise.

So in the first set of specs, I register an azure subscription with the service. The second set (Application) relies on having an azure subscription that's bee registered - so they need to run next. Once again, these tests exercise the 'application' APIs. Then there are another set of tests which depend on a valid 'Application' being available. The set ups are pretty costly - for ex the 'Application' tests provision infra on azure which can take a good bit of time... so doing it as part of the tests is what I've gone with. Downstream tests basically just check a few preconditions in before_all and fail if something isn't quite right.

In my scenario, fail fast probably makes sense since it wouldn't (in most cases) make sense to proceed on failure since tests aren't really isolated.

For this, I need to have my tests run in a deterministic order - the orderBy of spec name takes care of that. I'd imagine it doesn't hurt others as well.

[amirrajan]

I register an azure subscription with the service. The second set (Application) relies on having an azure subscription that's bee registered

Can't the be a nested context? Example:

class describe_api : nspec
{
  void registering_api()
  {
    act = () => ///
    it["works"] = () => //
    context["calling api"] = () =>
    {
      it["api call 1"] = () => //
      it["api call 2"] = () => //
    } 
  }
}

[amirrajan]

I'd imagine it doesn't hurt others as well.

The current NSpec behavior is to run specs in the order they were defined (it would affect the console output/there may have been some logical reasoning for defining the specs in that specific order that would be lost if we sorted by name).

[amirrajan]

Here is another example of nested contexts that may give you some inspiration: https://github.com/amirrajan/Oak/blob/master/Sample%20Apps/BorrowedGames/BorrowedGames.Tests/Controllers/describe_HomeController.cs#L69

This example also shows how reordering tests by name could cause some loss in documentation readability.

[raghur]

@amirrajan - I may be misunderstanding - but the only change I introduce is that the spec types are ordered by type name - the only (trivial) change is

 -            var specClasses = finder.SpecClasses();
+            var specClasses = finder.SpecClasses().OrderBy(type => type.Name);

this should not affect how test methods inside a class are run at all and does not touch anything about how nested contexts are run

[raghur]

To answer your question on nested contexts, I do use them... But typically, I want to keep each rest resource in its own test class to an extent

[BrainCrumbz]

The actual ordering of collected spec classes comes basically from Reflector:

return Assembly.LoadFrom(dll).GetTypes();
// no particular sorting set here, just what Assembly implementation returns

then from SpecFinder.SpecClasses:

finalList.AddRange(leafTypes);
// leaf types only are appended in same order as found by Reflector

then from SpecFinder.BaseTypes:

while (currentType != null)
{
    types.Add(currentType);
    currentType = currentType.BaseType;
}
// for each leaf type, base types are found in reverse order, 
// from derived to parent back up the chain

then from SpecFinder.SpecClasses again:

finalList.AddRange(BaseTypes(leafType));
// those base types will then be appended as found
// and finally made distinct, so some of them will be removed here and there

So it seems like there's not a big concern on actual spec class ordering. One could as well decide some particular ordering, and stick it into SpecClasses().

Just our 2c.

[amirrajan]

Aside: Fairly trivial to make your own NSpec runner.

https://gist.github.com/amirrajan/7ee4777788e6a5d76fee

Quick update, I'm trying your example locally to see what the test output looks like so we have a good example.

[amirrajan]

Yea, I was being dumb. Everything looks good. 🎉

[amirrajan]

fix is in 1.0.7 deployed to nuget.