python-hospital / hospital

Tools to diagnose Python projects (supervision, monitoring).
Other
40 stars 8 forks source link

Is it possible to customize the output of WSGI app from within a testcase? #71

Open benoitbryon opened 10 years ago

benoitbryon commented 10 years ago

See https://twitter.com/bartstroeken/status/467564038025932800

benoitbryon commented 10 years ago

What do you expect? What would you like to customize? The error messages? The JSON structure? Something else? Can you give an example?

bartee commented 10 years ago

Hi Benoit,

Thanks for responding, I thought about overwriting the TestResultResponse class, but there might be a better way. I'd like to be able to annotate the details of a testcase, for some extra info like the host where that specific test was executed on - in case of an external service or server.

By adding that, you can interconnect different hosts running hospital tests for example.

{
    "status": "pass",
    "details": [
        {
            "test": "test_http_200 (healthchecks.exampletest.DocumentationHealthCheck)",
            "status": "pass",
            "domain": "http://api.nu.nl", 
        },
        {
            "test": "test_connection (healthchecks.mysqltest.MySQLTestCase)",
            "status": "pass",
            "host": "192.168.1.44"
        },
        {
            "test": "test_ping (healthchecks.redistest.RedisHealthCheck)",
            "status": "pass",
            "host": "192.168.1.55"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 3,
        "expected_failure": 0,
        "error": 0,
        "fail": 0,
        "total": 3,
        "unexpected_success": 0
    }
}
benoitbryon commented 10 years ago

There may be some improvements around the context data passed to the AssertionError. As an example, at the moment, with a fail we get an additional "context" list:

{
    "status": "fail",
    "details": [
        {
            "test": "test_false (foo.bar)",
            "status": "fail",
            "context": "(<type 'exceptions.AssertionError'>, AssertionError(\"Failed to fetch URL http://localhost:8000/.\\nException was: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: / (Caused by <class 'socket.error'>: [Errno 111] Connection refused)\",), <traceback object at 0x7f34ac5bde18>)"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 0,
        "expected_failure": 0,
        "error": 0,
        "fail": 1,
        "total": 1,
        "unexpected_success": 0
    }
}

Note: the traceback in "context" is not really readable, see #72. But here is one idea:

... but the fact is I'm not sure how we can easily do this. Using the assertion message? assert True is False, "Here some information, but not easy to convert as a dictionary :("

If you had a (nice) way to populate such a context, would it be ok for you?

benoitbryon commented 10 years ago

A second workaround could be related to the execution context running the healthchecks. I mean, in your use case, the "host" information is shared by all the healthchecks run on a machine. At the moment, I have no nice idea about how we could do it. Perhaps using logging, but I'm not sure it would be easy to have relationships between an AssertionError and a log that was emitted before. What about working on #22 first, then see if it could be used or adapted to your use case?

benoitbryon commented 10 years ago

you can interconnect different hosts running hospital tests for example.

This is a story on its own! At the moment, I would tend to do something like that:

With such an architecture, on the "supervisor" (how would you name it?) you could easily have a result like that:

{
    "status": "pass",
    "details": [
        {
            "test": "GET http://api.nu.nl/healthchecks returns HTTP 200 OK",
            "status": "pass"
        },
        {
             "test": "GET http://192.168.1.44:1515 returns HTTP 200 OK",
            "status": "pass"
        },
        {
            "test": "GET http://192.168.1.55 returns HTTP 200 OK",
            "status": "pass"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 3,
        "expected_failure": 0,
        "error": 0,
        "fail": 0,
        "total": 3,
        "unexpected_success": 0
    }
}

If we had some hospital.assert_external_hospital(url) method, we could do it easily and have a better message in case of error, because it could take advantage of the JSON output. And it could highlight the information about external host (IP, port...).

Would this fit your use case?

That said, I also think the behaviour of such a "supervisor" should be:

I think we should have at least two levels of feedback:

With this idea in mind, I am not sure, at the moment, that a "supervisor" has to be able to concatenate results of every nodes, i.e. display details of each node. Whereas I think it has to be able to summarize results of each node, i.e. display fail/pass status of each node. If you need details about one node, isn't asking the node enough?

benoitbryon commented 10 years ago

And also, notice that we already can attach a custom message to assertions, but it is not easy to parse in the JSON output (it is a string, not a dictionary). assert expression, "your custom message". Perhaps it is enough, at least until we have better options.

benoitbryon commented 10 years ago

@bartee: the comments/questions above are meant to have a better idea about your needs, so that we can focus on what really matters ;) Tell me if I am misunderstanding or forgetting some point!

bartee commented 10 years ago

Wooah, you've been busy! :+1:

Interesting thoughts! After reading all this, I've gotten my hands dirty myself, and solved it for my usecase. First, I modified my testcase a bit, providing details:

@hospital.healthcheck
class RedisHealthCheck(unittest.TestCase):
    """
    Test the current state of Redis
    """
    def test_ping(self):
        r = redis.Redis(host=redis_host, port=redis_port,password=redis_password)
        try:
            r.ping()
        except redis.ConnectionError, e:
            self.fail(e)

    @property
    def details(self):
        return {
            'host': redis_host,
            'port': redis_port,
            'password': redis_password
        }

After that, I've added two lines to the details-method of the HealthCheckApp- class:

class HealthCheckApp(HealthCheckProgram):

    def details(self, result):
        """Generate report details dict."""
        for status, test, context in result.results:
            report = {
                'test': self.get_test_title(test),
                'status': status,
            }
            if context is not None:
                report['context'] = u'{context}'.format(context=context)

            if hasattr(test,'details'):
                report['details'] = test.details

            yield report

That works like a charm. In that way, the supervisor can use those details.details of a test case to interconnect different hosts. And you don't have to change the signature of existing output (or parse a string into something you can work with :-))

Only downside I can see: I can imagine details.details being a bit of a confusing name. But on the other hand, there might be use cases out there where other kinds of test specs are necessary.

I've already built a little supervisor-project myself. It's config based, checks a configured set of hosts for hospital-based output and puts that output into Redis. I yet have to plug it into my hosts-dashboard. Somewhere soon I'll put that little thing on my github.

bartee commented 10 years ago

About that supervisor: https://github.com/bartee/hospital_supervisor