Make test command more resilient to missing tooling

[koobs]

As an OS packager/porter, I'm not necessarily interested in pep/flake style development testing (though I will fix things and report upstream if I can).

While these of course are highly relevant to the author for people contributing code back to the project and for QA'ing PR's, I'm mostly interested in assuring quality for our end users by ensuring the test suite passes, and reporting failures back

Accordingly, I'd like to see the test command fail gracefully and continue in the face of ImportErrors for all the relevant steps in the test process. This also saves you (us) from having to create a special 'tests-only' command, which runs nothing but the tests (and feels a little backward)

Ideally, a special Makefile to coordinate these things is not necessary in the medium term, and rauth can use standard pythonic command classes to do everything it needs.

[maxcountryman]

Am I misunderstanding, or doesn't make test already do this?

[koobs]

setup.py nosetests runs tests, setup.py test runs:

python setup.py test
Checking pep8 compliance...
pep8: No such file or directory
*** [pep8] Error code 1

Then after installing pep8:

Checking pep8 compliance...
Running pyflakes...
pyflakes: No such file or directory
*** [pyflakes] Error code 1

Then after installing pyflakes:

Checking pep8 compliance...
Running pyflakes...
Running all tests...
No yanc plugin for nosetests found. Color output unavailable.
Running tests...
./run-tests.sh: 4: Bad file descriptor

FAILURE
*** [test] Error code 1

I stopped after that :)

I'm thinking:

• python setup.py test runs just the units tests (you can use test_suite = 'nose.collector' in setup.py)
• python setup.py check (or a suitable other name) does "all the things"
• That the check command (or equiv) doesn't fail fatally (ie; continue) if a tool isn't installed or raises ImportError
• So you don't need a Makefile (since this is a python project with a great test tool ecosystem)

Make things as simple as possible, but not simpler -Albert Einstein

[maxcountryman]

Yeah I think you should be using make test. :) It handles executing run-tests.sh, which requires setting some environment variables if you want to avoid things like pep8. We don't support setup.py test directly anymore; prefer make instead.

[koobs]

make test gives me:

No yanc plugin for nosetests found. Color output unavailable.
Running tests...
./run-tests.sh: 4: Bad file descriptor

FAILURE
*** [test] Error code 1

May I ask why the preference for make? I've seen a good push in other projects to standardise on setup.py test (with and without tox, py.test nosetest, support)

btw: Oh wow I just saw run_tests.sh's contents :)

[maxcountryman]

I can't reproduce that bad file descriptor error. Can you give us some more details about your environment?

Python's build tooling is just not a good match for what we're doing here, make is a good candidate and well supported across the board, plus I've used make several times with other Python projects and I find it's just a great tool all around. Going forward, we intend to use it.

[koobs]

No argument here, I just noticed it as an outlier in Python projects. I can continue to invoke python setup.py nosetests without kicking up anymore of a fuss :)

[maxcountryman]

Well I'd like to know why the test runner isn't working for you. make test should "just work". You can just use nosetests directly, no need to involve setup.py at all, if you go that route.

[koobs]

Happy to help if you can assist with debug or what/how I should be investigating. Am on IRC if you need me :)

[maxcountryman]

@koobs which IRC network?

[koobs]

Ah, freenode :)

[jamesjieye]

I had the same issue when I tried to run the test for the first time. And it turned out that the following libraries need to be installed first.

pip install pep8 pip install pyfakes pip install nose pip install coverage pip install mock

[maxcountryman]

@yejienihao that is documented in the README.