How to test parallel requests (race conditions)

[blueyed]

I wonder what is the best way to test race conditions with parallel web requests, where each request runs in its own transaction typically, but if both would insert the same data it would trigger an IntegrityError.

I've came up with using live_server and asyncio/aiohttp for it:

def test_user_ephemeralkey_race(active_requests_mock, drf_renter, renter, org,
                                live_server, event_loop, mock_stripe_customer):
    import asyncio

    import aiohttp
    import rest_framework_jwt

    # …

    url = '%s%s' % (live_server.url, reverse('app:name'))

    event_loop.set_debug(True)

    async def get_url():
        async with aiohttp.ClientSession(headers={
            'Accept': 'application/json; version=1.0',
        }) as client:
            async with client.get(url) as resp:
                text = await resp.text()
                assert resp.status == 200, text
                assert json.loads(text) == expected_response

    coros = asyncio.gather(
        get_url(),
        get_url(),
        loop=event_loop,
    )
    event_loop.run_until_complete(coros)

It works for me, but I wonder if there's a lighter solution that would not involve using transactional_db (via live_server) in particular.

If this makes sense we might want to add it to the documentation probably?

[blueyed]

It might be more lightweight to just call the method itself in an asyncio loop, but it's difficult (or even impossible?) to add the necessary (asyncio.)sleep in there.

[TauPan]

In a project using python 3.4 (no async syntax) I have a couple of testcases reproducing race conditions that use multiprocessing.Pool and/or Process together with a LiveServer monkeypatched to use ThreadedWSGIServer instead of the regular one (otherwise all servers use the same database connection, and/or don't really answer requests concurrently).

I'm wondering if the regular testcases.WSGIServer not being threaded wasn't an issue for you. Iirc this only was an issue in one testcase which checked a server-side race condition. (One testcase even had to monkeypatch or spy part of the server-side workflow to insert an assertion in the correct spot.)

(I'd have to dig up and sanitize the code if you want to see it. I haven't worked on that codebase for quite some time now.)

Those things get complicated (and slow) pretty quickly and I usually don't want to run these tests as part of the regular workflow. I usually mark them "slow" or something like that and only run them before doing releases.

Also those tests tend to cause problems because of #595 and/or #286.