tox.ini for locally run automated tests with multiple python versions

[vlcinsky]

Currently the package has definition of integration tests by means of .travis.yml. This allows for automated testing after changes are committed, however, a developer cannot run such tests locally without travis service.

Proposed solution (keep it simple)

• create tox.ini with definition of environments for all target python versions. Such environments would be run by default (would be listed in top section).
• each environment would run all the tests, which are easy to handle. Candidates for tests to skip are all tests requiring access to Internet or other installed services (like Mongo).
• .travis.yml would be kept as it is now as it runs more extensive set of tests.

Possible extensions of proposed solution

• drive even extended tests (requiring access to Internet or using external services like Mongo) by tox.ini. Possibly allow skiping such tests, e.g. by means of some env. variables or similar methods.
• modify .travis.yml to use what extended tox.ini is providing (I have seen couple of such examples). This way automated and local tests could be the same (or very similar).

If such idea sounds acceptable, I could provide PR with initial Proposed solution, which would conclude this issue.

Extended solution may follow, but it could include more work and extend the basic functionality too far blocking possible simplification of locally run tests for developer.

[amol-]

I'm fine with introducing tox, it's something that has been on my mind for quite some time, but couldn't find the time to do so.

I think btw that everything should go through tox, travis too. It doesn't make much sense to keep around two different testing configurations and tox can easily be driven by a travis through TOXENV.

If you want to submit a patch in that direction I'll gladly accept it.

[vlcinsky]

Great. I will put it into my task list. I agree, that .trafis.yml shall reuse what tox.ini will define.

My plan is to kick off the use of tox.ini and go as far as is easy for me, handing the rest (like more complex test cases involving other systems around) to you or to later time.

[vlcinsky]

Creating tox.ini and related test_requirements.txt file I see review of supported python versions is needed. Here is my checklist stating proposed python version support:

• python 2.6: keep it: great to have it for some users, skip it if in future difficulties arise
• python 2.7: keep it
• python 3.0: do not support: no support declared, do not support
• python 3.1: do not support: no support declared, do not support
• python 3.2: do not support: .travis.yml does not test this version, but setup.py contains dependency checks for this version
• python 3.3: keep it: .travis.yml tests this version
• python 3.4: keep it: .travis.yml tests this version
• python 3.5: keep it: .travis.yml tests this version
• python 3.6: add support: .travis.yml does not test this version (yet)

What shall be done:

• agree on list of supported python versions
• in setup.py declare supported explicit versions using trove classifiers
• in setup.py remove code used for not supported versions (like test dependencies for python 3.2)

[vlcinsky]

Shall we allow [testing] extras in setup.py?

Currently setup.py allows for installing with [testing] extras to run package based self-test.

I feel that such testing is obsolete now for couple of reasons and propose:

• define test dependencies by means of test_requirements.txt file (to be used with tox.ini)
• if you agree, I would remove [testing] extras from setup.py to keep it simpler. If you do not agree, I keep it present.

[vlcinsky]

Skipping locally run tests dependent on mongodb availability

$ tox

runs all tests happily if mongodb is running. However, with mongodb missing it fails.

I think, it would be better to skip such tests as is done for boto and AWS S3 credentials. As I am not so much experienced with nosetests fixtures, I would appreciate help with skipping tests in tests/test_storage_interface.py.

[amol-]

Merged in tox pull request and added skipping of mongodb tests when mongodb is not running.