Additional control over test order

[exodist]

I have been having a conversion via email about a desired feature. The requester has a very specific test setup where some things must run in a specific order, with other tests running parallel between them.

Currently they use the following yaml file to specify the run order when using prove.

# The wilrcards below ARE NOT regexes, they are globs and
# specific to TAP::Parser::Scheduler. The only possibilities are:
#    ** is any number of characters, including /, within a pathname
#    * is zero or more characters within a filename/directory name
#    ? is exactly one character within a filename/directory name
#    {foo,bar,baz} is any of foo, bar or baz.
#    \ is an escape character
seq:
  - seq:
    - par:
      - t/*
      - xt/{0,1,2,3}*
  - seq:
    - xt/40-dbsetup.t
    - par:
      - xt/40-database.t
      - xt/41-coaload.t
      - xt/45*.t
      - xt/47*.t
      - xt/48*.t
  - seq:
    - par:
      - xt/{43,44,5,7}*.t
      - xt/6{1,2,3,7,8,9}*.t
      - seq:
        - xt/60-startlsmb.t
        - seq:
          - par:
            - xt/6{5,6}*.t
        - seq: **.feature
      - xt/*.pg
  - seq: xt/89-dropdb.t

Achieving this behavior requires 2 things:

1. A custom Test2::Harness::Finder subclass
2. Custom behavior for Test2::Harness::Runner::State::advance_tasks()

I think a new plugin RulesYAML should be added, it can replace the default finder simply enough. Currently there is no way to specify a custom State subclass, but that functionality should not be difficult.

https://metacpan.org/pod/TAP::Parser::Scheduler has a description of the rules used when parsing the yaml file.

[exodist]

After re-reading the TAP::Parser::Scheduler, a custom finder is necessary. Those docs show that any tests not listed in the yaml will be run after everything else. So we probably only need a custom ::State subclass.

[toddr]

We have a less complex thing where we want t/zz_sanity.t to run at the end. The purpose of the test is to simply assure that none of the tests broke the environment unexpectedly. In our case, I considered using a category to accomplish this. HARNESS-CATEGORY-LAST. Not sure if that'd help here given the complexity but I thought I'd share the idea.

[exodist]

@toddr That may be useful, but would not solve the request in this ticket. This ticket has a small targeted demographic, and is essentially a "compatibility with prove" ticket for people already relying in a rules yaml file.

[ylavoie]

Eventhough we totally agree that tests have to be independent, the main reason we need the above setup is that xt/40-dbsetup.t validates that it can create a new database that is required for every test in the xt/40-79 range. Recreating the database for each test would be much too expensive in time and ressources to be a viable solution, so we had to come to that solution. After all tests are done, xt/89-dropdb.t validates that the database can be dropped properly.

We tried adding a setup/teardown but there was no way to make that available for a group of tests.

Moreover, adding the suggested custom modules allows complete compatibility with prove to ease migration.

[exodist]

I have been pondering this further. I think a plugin is overkill. I think ::State can be modified to order tests based on the yaml file and then move on to what it does normally for all other tests, which seems to be how prove handles the file as well. A flag can be added to the runner options to specify the yaml file.

@ylavoie does prove ALWAYS run any/all tests in the yaml file? If you ran prove and provided it a single test file instead of having it search for tests would it run just that test file, or everything in the yaml file? IE, does the yaml file tell prove what to run, or only what order to run things in?

[ylavoie]

@exodist prove only runs the tests that the user selected on the command line and defaults to all if none are provided. In all cases, they are run in respect of testrules.yml if one is available. testrules is only for sequencing, not selecting.

[ylavoie]

We managed to get the startup and shutdown issues out of the way, so for us the priority for that issue went down to 0. It could still be usefull to support testrules.yml to provide a drop-in replacement for prove though.