Consider per-suite pre & post executables

[cdent]

Like fixtures, but a call to an external executable, for when gabbi-run is being used.

This could be explicit, by putting something in the yaml file, or implicit off the name of the yaml file. That is:

• if gabbit is foo.yaml
• if foo-start and foo-end exist in the same dir and are executable

either way, when the start is called gabbi should save, as a list, the line separated stdout, if any, it produced

and provide that as args (or stdin?) to foo-end

this would allow passing things like pids of started stuff

/cc @FND for sanity check

[FND]

I would definitely prefer something explicit; anything else invites debugging headaches.

However, I don't fully understand your explanation of the behavior there.

[cdent]

I would definitely prefer something explicit

me too, just listing the other option for completeness

The behavior:

Given something like the following two yaml files:

a.yaml

executors:
- start: starter.sh
- end: ender.sh

tests:
- name: get the thing
   GET: /thing

b.yaml

tests:
- name: get the thing
   GET: /thing

and gabbi-run http://example.com/ -- *.yaml

When a.yaml was "run", first starter.sh would execute and perhaps start the server that hosts example.com. In its output would be the pid of that server, which gabbi would save in memory.

Then the tests would run.

Then ender.sh would run, passed the saved things. If it was a pid, it could be used to kill the server.

Other cleanups might be removing items from a database.

All the sorts of things one might do in a fixture.

When b.yaml is run, no starter.sh or ender.sh would be run.

[FND]

Ah, gotcha - so effectively, start's STDOUT is piped into end's STDIN. A few notes:

• using CLI arguments instead of STDIN there seems more complex and even brittle, even though it might be more convenient for the end script
• before/after seems more common? (as in established convention, unless you wanna go with setup/teardown)
• starting a server might be awkward because that process doesn't actually terminate - so that script would have to do its own process management (nohup or whatever) and it would be gabbits' responsibility to start with polling until the server is available?

[cdent]

The design and names need some work (I agree that before after is better), but the main question is the principle useful?

My thinking is that while there will be some awkwardness and requirement on the "executors" to sometimes be clever, it's no more than one would need otherwise if scripting gabbi-run into something with before and after handling, with benefit of more easily being per suite.

[FND]

If this doesn't save us the usual awkwardness (which I agree would be tricky to achieve), how is this better than just using a shell script (before; gabbi-run; after): because it's more granular, allowing for gabbits-specific scripts?

[cdent]

1. yes, gabbit specific
2. but also I was being somewhat flip about the awkwardness: gabbi itself provides a mechanism for sharing information between before and after which means that pids, temp files, db identifiers, etc are easier to transmit. It's not a huge improvement in awkwardness but it is a bit.

Also, in a CI situation, there's some usefulness to being able to have/see the before and after right next to the gabbits.

I dunno, at this stage it is just an idea that came to me in the shower, so I thought I would try to tease it out.

[FND]

You've convinced me: I do believe this could be useful. (Despite my questions, I didn't need much convincing in the first place.)