CleanCut / green

Green is a clean, colorful, fast python test runner.
MIT License
793 stars 75 forks source link

pass with unittest/fail with green #146

Closed sjsadowski closed 7 years ago

sjsadowski commented 7 years ago

I've been using green as a test runner for awhile now, but at this point I'm not sure what I'm doing wrong or if this is an actual bug. My assumption is that this is my issue, but my google-fu has failed. I'm also operating under the idea that the test passes without using green, but fails in green.

using python 3.5, green 2.5.2 & falcon 1.1.0

very basically, when doing test running with straight unittest2, the tests pass. When running with green, the tests fail. See comparisons below

Unittest2:

$ python -m unittest2 -v test/test_app_01.py 
test_default_get (test.test_app_01.AppTest01) ... ok
test_default_get (test.test_app_01.GroupTest01) ... ok
test_default_get (test.test_app_01.UserTest01) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK

green:

$ green -d -v
2016-12-20 11:17:56     DEBUG Attempting to load target '.' with file_pattern 'test*.py'
2016-12-20 11:17:56     DEBUG Test case names: ['test_default_get']
2016-12-20 11:17:56     DEBUG Test case names: []
2016-12-20 11:17:56     DEBUG Test case names: ['test_default_get']
2016-12-20 11:17:56     DEBUG Test case names: ['test_default_get']
2016-12-20 11:17:56     DEBUG Load method: DISCOVER - .
2016-12-20 11:17:56     DEBUG Found 3 tests for target '.'
2016-12-20 11:17:56     DEBUG Attempting to load target 'auth.test.test_app_01' with file_pattern 'test*.py'
2016-12-20 11:17:56     DEBUG Load method: DOTTED OBJECT - auth.test.test_app_01
2016-12-20 11:17:56     DEBUG Found 3 tests for target 'auth.test.test_app_01'
FFF

Failure in auth.test.test_app_01.AppTest01.test_default_get
  File "/usr/lib64/python3.5/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib64/python3.5/unittest/case.py", line 617, in run
    testMethod()
  File "/projects/services/auth/test/test_app_01.py", line 18, in test_default_get
    result = self.simulate_get('/')
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 594, in simulate_get
    return simulate_get(self.app, path, **kwargs)
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 374, in simulate_get
    return simulate_request(app, 'GET', path, **kwargs)
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 333, in simulate_request
    iterable = validator(env, srmock)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 153, in lint_app
    check_environ(environ)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 336, in check_environ
    check_errors(environ['wsgi.errors'])
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 373, in check_errors
    % (wsgi_errors, attr))
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 128, in assert_
    raise AssertionError(*args)
AssertionError: wsgi.errors (<green.output.GreenStream object at 0x7f5c1e6c7f60>) doesn't have the attribute writelines

Failure in auth.test.test_app_01.GroupTest01.test_default_get
  File "/usr/lib64/python3.5/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib64/python3.5/unittest/case.py", line 617, in run
    testMethod()
  File "/projects/services/auth/test/test_app_01.py", line 30, in test_default_get
    result = self.simulate_get('/user')
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 594, in simulate_get
    return simulate_get(self.app, path, **kwargs)
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 374, in simulate_get
    return simulate_request(app, 'GET', path, **kwargs)
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 333, in simulate_request
    iterable = validator(env, srmock)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 153, in lint_app
    check_environ(environ)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 336, in check_environ
    check_errors(environ['wsgi.errors'])
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 373, in check_errors
    % (wsgi_errors, attr))
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 128, in assert_
    raise AssertionError(*args)
AssertionError: wsgi.errors (<green.output.GreenStream object at 0x7f5c1e6c7b70>) doesn't have the attribute writelines

Failure in auth.test.test_app_01.UserTest01.test_default_get
  File "/usr/lib64/python3.5/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib64/python3.5/unittest/case.py", line 617, in run
    testMethod()
  File "/projects/services/auth/test/test_app_01.py", line 24, in test_default_get
    result = self.simulate_get('/user')
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 594, in simulate_get
    return simulate_get(self.app, path, **kwargs)
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 374, in simulate_get
    return simulate_request(app, 'GET', path, **kwargs)
  File "/projects/services/auth/auth_venv/lib/python3.5/site-packages/falcon/testing/client.py", line 333, in simulate_request
    iterable = validator(env, srmock)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 153, in lint_app
    check_environ(environ)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 336, in check_environ
    check_errors(environ['wsgi.errors'])
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 373, in check_errors
    % (wsgi_errors, attr))
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 128, in assert_
    raise AssertionError(*args)
AssertionError: wsgi.errors (<green.output.GreenStream object at 0x7f5c1e6c7cc0>) doesn't have the attribute writelines

Ran 3 tests in 0.120s

FAILED (failures=3)
dougthor42 commented 7 years ago

Just FYI, use triple-backticks (```) for code blocks. I've updated your post.

Can you link to a project or post the code that shows this issue?

sjsadowski commented 7 years ago

Hey, no problem. Wasn't paying attention and just hit the 'insert code' button without thinking.

Abbreviated code/tests here. All the current tests that have this issue are the same. Let me know if you'd like anything else.

App code: (./app.py)

import json
import falcon

class DefaultResource(object):
    def on_get(self, req, resp):
        resp.status = falcon.HTTP_200
        resp.body = json.dumps({"response": resp.status})

app = falcon.API()

defaultresource = DefaultResource()

app.add_route('/',defaultresource)

Test code: (./test/test01.py)

from falcon import testing
import falcon
import json
import app

class AppTestCase(testing.TestCase):
    def setUp(self):
        super(AppTestCase, self).setUp()
        self.app = app.app

# Basic test to see if functionality works. HTTP GET / should always be 200 OK

class AppTest01(AppTestCase):
    def test_default_get(self):
        doc = json.dumps({"response": falcon.HTTP_200})
        result = self.simulate_get('/')
        self.assertEqual(json.dumps(result.json), doc)

output:

$ python -m unittest -v test/test01.py 
test_default_get (test.test01.AppTest01) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
$ green -v -d
2016-12-20 13:25:44     DEBUG Attempting to load target '.' with file_pattern 'test*.py'
2016-12-20 13:25:44     DEBUG Test case names: ['test_default_get']
2016-12-20 13:25:44     DEBUG Test case names: []
2016-12-20 13:25:44     DEBUG Load method: DISCOVER - .
2016-12-20 13:25:44     DEBUG Found 1 test for target '.'
2016-12-20 13:25:44     DEBUG Attempting to load target 'test.test01' with file_pattern 'test*.py'
2016-12-20 13:25:44     DEBUG Load method: DOTTED OBJECT - test.test01
2016-12-20 13:25:44     DEBUG Found 1 test for target 'test.test01'
F

Failure in test.test01.AppTest01.test_default_get
  File "/usr/lib64/python3.5/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib64/python3.5/unittest/case.py", line 617, in run
    testMethod()
  File "/projects/services/falcon_testing/test/test01.py", line 16, in test_default_get
    result = self.simulate_get('/')
  File "/projects/services/falcon_testing/venv/lib/python3.5/site-packages/falcon/testing/client.py", line 594, in simulate_get
    return simulate_get(self.app, path, **kwargs)
  File "/projects/services/falcon_testing/venv/lib/python3.5/site-packages/falcon/testing/client.py", line 374, in simulate_get
    return simulate_request(app, 'GET', path, **kwargs)
  File "/projects/services/falcon_testing/venv/lib/python3.5/site-packages/falcon/testing/client.py", line 333, in simulate_request
    iterable = validator(env, srmock)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 153, in lint_app
    check_environ(environ)
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 336, in check_environ
    check_errors(environ['wsgi.errors'])
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 373, in check_errors
    % (wsgi_errors, attr))
  File "/usr/lib64/python3.5/wsgiref/validate.py", line 128, in assert_
    raise AssertionError(*args)
AssertionError: wsgi.errors (<green.output.GreenStream object at 0x7f1adfd29438>) doesn't have the attribute writelines

Ran 1 test in 0.123s

FAILED (failures=1)
CleanCut commented 7 years ago

@sjsadowski By default, green replaces Python's sys.stdout and sys.stderr objects with GreenStream objects so that the output can be intercepted and presented more nicely. Looks like your code is trying to call .writelines() on one of those...which we haven't implemented.

Quick fix: Use --allow-stdout

-a, --allow-stdout    Instead of capturing the stdout and stderr and
                      presenting it in the summary of results, let it come
                      through.

Better fix:

diff --git a/green/output.py b/green/output.py
index 2ad9750..cfc110e 100644
--- a/green/output.py
+++ b/green/output.py
@@ -157,6 +157,13 @@ class GreenStream(object):
             text = text_type(unidecode(text))
         self.stream.write(text)

+    def writelines(self, lines):
+        """
+        Just for better compatibility with real file objects
+        """
+        for line in lines:
+            self.write(line)
+
     def formatText(self, text, indent=0, outcome_char=''):
         # We'll go through each line in the text, modify it, and store it in a
         # new list
sjsadowski commented 7 years ago

Thanks @dougthor42 @CleanCut - as I suspected, PEBKAC.

CleanCut commented 7 years ago

The patch is included in version 2.5.3. I just released it.