pytest-httpx 0.32 and stamina

[gsakkis]

I am using stamina and running http tests with stamina.set_testing(True, attempts=2). This used to work with pytest-httpx<0.32 but raises httpx.TimeoutException on v0.32. Now I can fix this with either setting the retry attempts to 1 or with the @pytest.mark.httpx_mock(can_send_already_matched_responses=True) decorator. What do you suggest? Any pros/cons for each approach? cc @hynek

[Colin-b]

Hello @gsakkis

I am not familiar with stamina, so my understanding might be wrong. It looks like a smart retry layer you can put on top of specific parts of your code.

I assume it only retries upon some criteria you provide, or maybe based on exception context detection. In any case, pytest-httpx default behavior is now to ask for the expected number of requests by requiring the same number of responses to be registered.

If you have test cases reproducing HTTP errors (such as non 2XX/3XX status code) or failures (exception raising such as a ConnectTimeout) and you have a retry mechanism in place, it all depends on what you want to test really.

If you want to retry X times with stamina (in your case once I guess, as first attempt + first retry = 2), then I would register 2 responses on the same HTTP call. The order of registration matters so the first one will be sent in response to the first request, and the second one will be sent in response to the retry of stamina.

I don't know what your test scenario is but you basically have 2 options here:

1. You want to test a retry that works -> First response is an error/failure, second response is a success
2. You want to test a retry that still fails -> First response is an error/failure, second response is also an error/failure.

You can register the same response more than once, it is not an issue, on the opposite, this will help maintenance of your code by explicitly stating what is expected to happen.

Note that this only works if you have a deterministic test case (ie: a test that will have the same behavior every time you run it) but I assume this is the case.

As stated in the documentation, the marker option to send already matched responses is to use with caution as you will then not know for sure how many requests where issued (unless you check them in your test case afterwards using httpx_mock.get_requests of course)

Let me know if you still need more feedback.

[gsakkis]

@Colin-b thanks for the reply!

If you want to retry X times with stamina (in your case once I guess, as first attempt + first retry = 2), then I would register 2 responses on the same HTTP call.

That's what I ended up doing, but since there are many tests like this I wrapped it in a context manager that registers N httpx_mock responses and then sets up stamina's testing for the context:

from contextlib import contextmanager
import stamina

@contextmanager
def retrying_httpx_mock(httpx_mock, attempts, **kwargs):
    for _ in range(attempts):
        httpx_mock.add_response(**kwargs)
    stamina.set_testing(True, attempts=attempts)
    yield
    stamina.set_testing(False)

And here's how it can be used to test a minimal stamina-wrapped httpx client:

import httpx
import pytest
import stamina

class RetryingClient(httpx.Client):
    @stamina.retry(on=httpx.HTTPError)
    def request(self, *args, **kwargs):
        resp = super().request(*args, **kwargs)
        resp.raise_for_status()
        return resp

def test_retrying_client(httpx_mock):
    with RetryingClient() as client:
        with retrying_httpx_mock(httpx_mock, attempts=3, status_code=500):
            with pytest.raises(httpx.HTTPStatusError) as excinfo:
                client.post("https://test_url")
            assert len(httpx_mock.get_requests()) == 3
            assert excinfo.value.response.status_code == 500

[Colin-b]

Thanks for the feedback.

You can get rid of your assert in the number of requests as httpx-mock will do this check as well at teardown, so your test case could look like the following:

def test_retrying_client(httpx_mock):
    with RetryingClient() as client:
        with retrying_httpx_mock(httpx_mock, attempts=3, status_code=500):
            with pytest.raises(httpx.HTTPStatusError) as excinfo:
                client.post("https://test_url")
            assert excinfo.value.response.status_code == 500