Ability to pass arbitrary testinfra | serverspec cli flags to `molecule verify`

[tknerr]

Akin to #572 it would be nice if you could pass arbitrary flags to testinfra or serverspec upon molecule verify

Just one example use case (but guess there are many more): some of my tests are marked with a specific tag or marker, and I'd like to run only these, e.g.:

$ # run only testinfra tests makred with xxx
$ molecule verify -- -m xxx

[retr0h]

I'd prefer not to go down this slippery slope. Since, I really don't want to implement such functionality in all subcommands. If you wish to do that, you can provide options to the verifier in molecule.yml.

verifier:
  name: testinfra
  options:
    n : 3

[tknerr]

@retr0h why a slippery slope? Actually, I'd rather see it as more consistent: if you can pass arbitrary args to the underlying config mgmt system when molecule convergeing, why should you be able to do that for the underlying test framework when molecule verifying?

It would probably be a power user feature only, but would be super useful for ad-hoc commands where you don't want to temporarily edit / modify the molecule.yml

[retr0h]

I'll think about it ;)

[tknerr]

Thanks @retr0h !

While this request is asking for a generic solution, there is actually a very concrete need / problem behind it:

For some stuff we have quite a bunch of tests, so while developing on a sepcific feature I'd love to run only a specific set of these tests. With molecule converge there is a --tags flags, so for provisioing this case is covered. For molecule verify there ain't no --tags yet, so you always have to run all the tests.

(also in pytest it's called markers not tags as far as I understand... but may be wrong as I' rather new in python land ;-))

[retr0h]

@tknerr this isn't something I see us implementing. The CLI usage would be very confusing.

1. Making the usage consistent across verifiers would also be difficult since serverspec and goss are so much different than testinfra.
2. Passing in verifier args to the CLI like we did in #572 would an odd UX, b/c the verifier runs various binaries. In the default case it runs flake8, testinfra, and ansible-lint. The UX would not be clean to know which binary we wish to pass the verify args to.

Due to these reasons, I prefer not to tackle this.

[azaghal]

Instead of opening up a new issue, I figured it might be better to add comment to this one - feel free to direct me towards opening up a new one :)

I was wondering if it would make sense to perhaps implement at least some way to limit the set of tests that would get run? Similar to original poster, I often find myself wishing I could run one or two tests only - usually the failing ones or new ones that I am working on. While modifying the configuration file is one way to do it, it's not quite optimal (you need to keep track of a "helper" modification and avoid committing it, you need to update it when switching to different test etc).

The main issue would be supporting all the different ways for limiting tests. Pytest/testinfra can be rather flexible in that regard, and no idea what kind of limiting could be done via Goss or InSpec.

@retr0h Would you still be categorically against adding some kind of mechanism like this to the CLI?

[decentral1se]

I would also love to have this. I do see the issues highlighted and do accept it could be a maintenance pain. Can we find a compromise? Here is my proposal - could we add a molecule verify --testinfraargs="--all-my-testinfra-args" --pytestargs="--all-my-pytest-args" (where the form is --<thing>args=...) as catch alls, single optional argument to pass things down the line to relevant places which avoid any potentially tricky CLI parsing logic? This would also allow users to deal with the different ways each verifiers accept args.

[azaghal]

Not sure tat adding capability to pass-in arbitrary arguments would help in terms of usability/ubiquitousness of available options, though. Still kind of interested to see if there is any sort of compromise that can be made prior to perhaps trying to tackle the issue :)

[retr0h]

I suggest just running the testinfra commands by hand. Run molecule --debug verify and grab the long env variable line and add the testinfra command executed. You can then twiddle the commands exactly as you like while you develop. Having Molecule handle every developers possible workflow is tricky. You can do what I described above with a couple cut and pastes.

[decentral1se]

While I do accept copy/pasta as a work around, surely we can come up with something less laborious.

I would not see this is as particular developer workflows (which sure, I can agree, we can't always cover) but more as access to the basic tools that molecule uses. Right now, it is just hard to configure how they run and I would guess all users have at least once searched for how to do this in the github issues and --help output.

PYTEST_ADDOPTS environment variable is currently how I pass arguments to pytest. Perhaps if we could get internal tools to also accept an environment variable for arguments (and therefore let the internal tool manage adding those arguments) then we it would be easier for molecule to pass them down. Still typing PYTEST_ADDOPTS="-k mysql -s -vv" molecule verify is a bit of a pain, when it could be molecule verify --pytestargs/--pta "-k mysql -s -vv" or something.

[azaghal]

While I generally agree one can't cover all the use-cases, I'd be somewhat surprised that people don't run into this kind of need more often - when running entire set of tests for a role can take 60 seconds or so, being able to pinpoint failing tests is very useful.

To try to take a bit of an orthogonal approach to this, given that I've seen now at least two alternatives, would it make sense to document the copy/paste way to run tests and/or the PYTEST_ADDOPTS variant (I actually did not know about it before now). Or has this already been fully covered in docs?

@retr0h Thanks for getting back to us in either case :)