Add support for setting discovery arguments via python module paths

[ygbourhis]

Issue description

This is more a feature wish than a bug:

Someone might need stestr to run tests from a pip installed package. However, these pip installed packages can be in /usr/local/lib/pythonX.X/dist-packages/ or in a virtual environment. And in a virtual environment, the directory in which are the tests can also have been installed by pip install -e somepackage/ when "somepackage" is a git repo.

Therefore, making a stestr.conf file which will work on unique platforms is currently impossible. when the tests scripts are in another module/repo than the repo you work on.

I.E.: I work on a repo of my openstack platform deplyment, and need to run tempest tests which are in a python module (tempest is a set of tests which run on a deployed openstack environment. I did not write these tests. tempest tests are there to check your openstack platform works properly).

Currently, I need to do this kind of stuff to make stestr platform agnostic so that anyone can launch our openstack tempest tests:

# create .stestr.conf:
TEMPEST_PATH=$(/usr/bin/env python -c "import os, tempest;print(os.path.dirname(os.path.realpath(tempest.__file__)))")
TEMPEST_TEST_DIR="$TEMPEST_PATH/test_discover"
echo "creating $(pwd)/.stestr.conf file"
echo "[DEFAULT]" > .stestr.conf
echo "test_path=$TEMPEST_TEST_DIR" >> .stestr.conf
echo "top_dir=$TEMPEST_PATH" >> .stestr.conf
echo 'group_regex=([^\.]*\.)*' >> .stestr.conf
# Now you can run stestr

We have a CI, so I can hard code the conf for our CI, but if one wants to launch a new set of tests (new tempest version) to check these tests work on the platform before pushing them to the CI, the .stestr.conf for the CI will not work.

It would be a very nice feature to be able to configure a "python module import path" instead of a directory.

Expected behavior and actual behavior

Curently we need this kind of configuration where someuser can be anyone:

[DEFAULT]
test_path=/some/path/which/may/be/different/to/my_package/test_discover
top_dir=/some/path/which/may/be/different/to/my_package
group_regex=([^\.]*\.)*

A very nice feature would be to be able to provide such configurations:

[DEFAULT]
module_test_path=my_package.test_discover
module_top_dir=my_package
group_regex=([^\.]*\.)*

Of course, it would work only if my_package.test_discover is in the python path. See in my example above how I find the real path from the python module. Python is totally capable of doing so (since I find the path with python commands). So having stestr supporting this would be more than great.

Steps to reproduce the problem

Create a .stesr.conf in your openstack platform deployment repo which works for you and allows you to launch tempest on the installed platform (tempest being a python module). Let someone clone your repo If this someone's $HOME path is not the exact same (and it's generally the case), stestr will fail unless this user recreates a .stestr.conf file.

Specifications like the version of the project, operating system, or hardware

Runing openstack tempest tests on openstack platforms.

System information

**stestr version (stestr --version): stestr 2.0.0

**Python release (python --version): Python 2.7.6 and Python 3.4.3

Of course I'm totally aware it's not a bug, but since python can very easily find a path from an importable python module, being able to configure a pyhton module for testing would be a very very nice feature.

Let me know if anything was unclear.

Regards.

[mtreinish]

So while I agree this feature request would be nice to have I do really wonder how much use it would really have. It seems like a pretty uncommon use case to want to run tests on a package that's already been installed outside of where that code lives. It also depends on tests being packaged in the sdist, which isn't a guaranteed pattern. But, either way it likely won't harm anything and shouldn't be hard to implement, I think it's a good low hanging fruit for a new contributor.

As an aside the use case you're describing here is actually something that tempest supports natively already. This is part of the concept of tempest workspaces (see step 3 in: https://docs.openstack.org/tempest/latest/overview.html#quickstart ) That will create a self contained environment for running tempest from an installed tempest package. While the guide there recommends using tempest run after workspace creation, you can use stestr directly. (since tempest run is just calling it internally)

[mtreinish]

I just updated the title to be a bit more reflective of what the feature request is.

[ygbourhis]

Thanks @mtreinish

To explain what we did is exactly as you mention in https://docs.openstack.org/tempest/latest/overview.html#quickstart :

tempest init our_platform1
tempest init our platform2
# etc...

And then for each platform we followed https://docs.openstack.org/tempest/latest/configuration.html#tempest-configuration And we made a git repo of all this which contains the platforms configurations and the tempest tests to whitelist/blacklist depending on the platform. Tempest and all openstack clients are in a python requirements.txt file of the repo, and after cloning the repo one just needs to pip install the requirements in a virtualenv and launch all tests or a specific test when he modified something on the platform, in order to check non-regression (we have a CI doing it automatically, but it's good practice that the operators of the platform check manually at ounce with specific tests targeted on the platform change before the CI launches all tests, just to revert fast in case of issues).

Now the issue with this is that tempest init created .testr.conf files with hard coded paths... Any one who clones our repo has therefore non working paths once cloned as you can easily understand, since the virtualenvs have different paths. So I'm making scripts to regenerate these .testr.conf files automatically, post git clone.

Once stestr accepts python module paths, what I'll do afterwards is raise a launchpad ticket on tempest (and os-testr) so that tempest init uses python module paths instead of hard coded paths when it generates the conf files (and so that it generates .stestr.conf files instead of .testr.conf files, but conversion is trivial).

Thanks a lot.