Async Step Definitions

[jeduden]

is there away to use async step definitions with pytest-bdd ?

For a step definition like:

@when('i send cucumbers ')
async def i_send_cucumbers(loop):
      pass

I see the warning: pytest_bdd/scenario.py:137: RuntimeWarning: coroutine 'i_send_cucumbers' was never awaited

[olegpidsadnyi]

what about async then and async given? does it even make sense?

[olegpidsadnyi]

will this work for you?

async def send_cucumbers(): pass

@when('i send cucumbers ') def i_send_cucumbers(loop): loop.run_until_complete(send_cucumbers)

[jeduden]

@olegpidsadnyi This works, however, in my case all step definitions are async (and can call multiple times into async functions). so this would mean a lot of calls to run_until_complete. As a work-around I have defined a decorator that schedules the function on the event loop.

A generic solution would be to have the ability to easily wrap any step definition with a function. (similar to the before and after hooks).

And then it would be great if support of async functions would be provided out of the box.

Especially, after reading the discussion about implicit and explicit eventloops in the thread here: https://groups.google.com/forum/#!msg/python-tulip/yF9C-rFpiKk/tk5oA3GLHAAJ seems to be tending towards implicit loops for python code that runs on 3.5.3+ and/or 3.6+.

So code that explicitly calls run_until_complete to run async functions will look more and more awkward.

[olegpidsadnyi]

@jeduden this means pytest-bdd has to depend on yet another library asyncio? I don't really see how a gherkin scenario can by async. If the whole point that it has not to break i think you need to extend your testing suite with some kind of support for async functions (decorator is a good idea).

I think semantically you can't really represent async process in step-by-step imperative Gherkin. Therefore it should be explicit. If you have to initiate few async messages - they should stand behind one When step that describes this exercise in a form that humans understand. Gherkin is not a programming language, it is the way to describe steps to humans which can't do async computation in their brains.

What do you think, @bubenkoff ?

[bubenkoff]

well, the async question comes down to the pytest, not specifically to pytest-bdd. And for pytest's dependency injection, it's not realistic that it starts to support async fixture definitions anytime soon. Also for tests, while it sounds cool there seems to be a little win to have async fixtures, simply because fixtures should be fast enough, and then it should not matter much if you optimise dependency graph in a way that you run fixtures in parallel for some parts of the graph. It worth effort though to add the 'async clause' to the documentation mentining the workaround to cut the async point on the fixture definition by waiting for async function to finish

[olegpidsadnyi]

@jeduden could you show your decorator implementation for when step that is awaiting?

[bubenkoff]

FYI: there's a helper apparently to minimize the efforts: https://pypi.python.org/pypi/pytest-asyncio

[jeduden]

Support on pytest side is fine. We are using aiohttp test-support and with pytest-asyncio 0.8.0 we can also the helpers from that package.

However, these helpers work on scenario level.

The step_definitions are one level below, and since the calling code here: https://github.com/pytest-dev/pytest-bdd/blob/master/pytest_bdd/scenario.py#L137 is not checking if step_func is async. i.e. it only supports synchronous step functions.

In order to fix this, we currently need to wrap all steps with decorators like this:

 def sync(func):
      @wraps(func)
      def synced_func(*args, **kwargs):
          loop = kwargs.get("loop")
          if not loop:
              raise Exception("Need loop fixture to make function sync")
          return loop.run_until_complete(func(*args, **kwargs))
      return synced_func

[jeduden]

@olegpidsadnyi declaring a function async doesn't mean it is not imperative. Async functions are imperative like synchronous functions. However, with await / yield from you explicitly define points where the execution of other scheduled coroutines is allowed.

In order to schedule the parallel execution of multiple asynchronous functions you would use helpers like asyncio.gather ( see https://docs.python.org/3/library/asyncio-task.html#example-parallel-execution-of-tasks )

The reason why we need to declare our step_definitions async is because we are using the aiohttp client inside to perform networking calls as part of our step definitions. Furthermore, the rest of the code base is completely async, hence we think it is only natural that also step_defintions are written with async.

[bubenkoff]

As i see from the pytest-asyncio code, it just awaits for every 'async' fixture, so there's no real parallelism possible between the fixtures And how do you mean pytest upports async fixtures in a 'proper' way? Can't really see that

[bubenkoff]

But i do see your point about the need of the wrapper everywhere - it sucks. Looks like the only way to avoid that is to depend on pytest-asyncio and use it's helpers directly in the pytest-bdd

[jeduden]

@bubenkoff even that is not possible since there is no hook that allows me to insert a custom decorator on the functions.

even if pytest-bdd doesn't support async step definitions out of the box. A hook for wrapping a step definitions would help to reduce duplication.

[bubenkoff]

but I meant to change pytest-bdd itself to support that automatically

[bubenkoff]

are you up for making a PR which will add pytest-asyncio as a dependency and automatically use it to resolve async step definitions, if they are async?

[jeduden]

@bubenkoff In general, I am happy to make a PR. One question regarding the pytest-asyncio dependency. You mention it because we need to have a library to provide the loop fixture, right ?

[bubenkoff]

@jeduden yes, also to keep as much core async stuff as possible in a single plugin (pytest-asyncio)

[jeduden]

@bubenkoff i did a quick check on the current test-suite; how i can best write some this for this new type of possible step functions. do you have a pointer for me where i best can add tests ? Or should i create a complete new file ?

[bubenkoff]

You can put a new file here tests/steps/test_async.py copying the approach of tests/steps/test_unicode.py and replacing definitions with async ones:

@given
async def ...

@when
async def ...

@then
async def ...

[s0undt3ch]

https://github.com/pytest-dev/pytest-bdd/pull/221

[vodik]

For what its worth, I took the approach with #221 and then added that hook implementation as a separate pytest plugin that I can install as necessary.

[Victor-Savu]

@vodik

added that hook implementation as a separate pytest plugin

Is your pytest plugin available anywhere, just so I can try it out? Do you plan on maintaining it? My gut feeling is that integrating the functionality into pytest-bdd would be the path of least maintenance burden, but that is up to the maintainers I suppose.

[Victor-Savu]

First off, thank you so much for this awesome library! I was hoping I can contribute my experience with asyncio.

@olegpidsadnyi sad here:

this means pytest-bdd has to depend on yet another library asyncio?

Depends on how you look at it. asyncio is part of the standard library since 3.4. So users wouldn't need to install anything special. This feature is not needed by users of older versions of python since they don't use the async/await syntax to begin with, so they have no need for it.

I think semantically you can't really represent async process in step-by-step imperative Gherkin.

Agreed, but the cool thing is that despite the name, async/await doesn't necessarily mean that the operations happen in arbitrary order. await just means that async operations suspend the current execution thread (not system thread, mind you) and execution is resumed once they are completed, in the same order as specified in the function. This way, the semantics of the operations are unchanged happen in a step-by-step imperative mode. Please let me know if you would like to have a conversation about this topic. I'm always happy to talk about async/await :)

[rgreinho]

Behave 1.2.6 added some decorators to Testing asyncio Frameworks .

That may be inspirational to implement something similar for pytest-bdd.

[pjz]

Is there a guide somewhere to using pytest-bdd with pytest-asyncio ? I'm not sure how to make them play nicely together and actually execute my bdd tests.

[DjaPy]

@bubenkoff Hi! Are you planning to close this PR?

[mathew-jithin]

Just shooting this question as I am stuck with the same issue for handling async step definitions as pytest-asyncio cannot be used here.

Is there some update or any way to handle the same as of now? Hope you will please help if some new changes/ways to handle are present.

@bubenkoff

[jruizaranguren]

Most of web development nowadays is moving to async frameworks (for instance fastapi). Testing this frameworks typically involves running async app and also use an async testclient. Support for async pytest-bdd step definitions would definitely help in further adoption in these kind of projects.

[bubenkoff]

@DjaPy not for me to decide, I've requested a review from @youtux

[DjaPy]

I forked the project and applied PR. I was able to take advantage of asynchronous tests. But as time passes, I see that this is unnecessary. If you write integration tests, then nothing prevents you from using requests as a client. I will support the voiced idea that adding asynchrony is redundant and solves other problem.

[detoyz]

@DjaPy but what about if your integration tests are part of your whole asyncio project with lots of async tests? Isn't it's better to just run pytest on the whole project and view all test results at once? I thought that's one of the key advantages of pytest-bdd. Otherwise, if you write your bdd tests fully isolated from other project test structure (async fixtures for example), what's the benefit then of using pytest-bdd instead of let's say behave?

Currently, I actually do as you suggested, by separating bdd tests (using requests in them) and other tests. However again, for local development, for CI/CD I always need to run two commands instead of one. I mean it's not probably a "must-have" feature, but definitely "nice-to-have" for pytest-bdd.

---

BTW want to give my warm thanks for an awesome library :rocket:

[DjaPy]

I thought that's one of the key advantages of pytest-bdd. Otherwise, if you write your bdd tests fully isolated from other project test structure (async fixtures for example), what's the benefit then of using pytest-bdd instead of let's say behave?

This is an argument. When you're used to writing in pytest and you don't want to dive into behave. I'll choose pytest-bdd. Yet here another problem is solved. The problem of infrastructure.

[limonkufu]

Do we have a workaround getting async to work with pytest-bdd with aiohttp with async with? I was wrapping my async calls with `loop.run_until_complete' as suggested above but it just means that I have to define every fixture twice and then wrap them just to get it to work.

[leferradkw]

Is there some update about this? It would be a great implementation, given there are many scenarios with async calls that would be wanted to test with this library.

In the meantime, what I'm doing is to use asyncio.run() calls on every step that have async functions, and it's working to me.

[youtux]

I made a POC at https://github.com/pytest-dev/pytest-bdd/pull/629. It only implements the execution of async step functions. If you need to use async fixtures, you can do that via a plugin like pytest-asyncio.

The implementation of the fix seems so trivial that I'm not sure I want to include it in pytest-factoryboy. Anybody can make a decorator that converts async functions to sync:

import asyncio, functools

def async_to_sync(fn):
    """Convert async function to sync function."""

    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        return asyncio.run(fn(*args, **kwargs))

    return wrapper

and apply it to their async steps:

@given("there is a cucumber", target_fixture="cucumber")
@async_to_sync
async def _():
    return 42

[leferradkw]

Nice work @youtux !! I think it would be more like a workaround to deal easier with async functions in the steps, but there should be a better way to actually support async instead of converting async calls into sync one. I'm not familiar with the architecture of this library so I can't really say, but it looks like a support that should exist without workarounds. In the meantime you solution really helps!

[Cito]

Thanks @youtux. This workaround was actually already suggested 6 years ago.

Unfortunately it is not compatible with using pytest-asyncio for running async fixtures. When the test function uses an async fixture via pytest-asyncio, then an event loop is already running, but the asyncio.run call will create a different event loop and not run the test code inside the already running event loop used with the async fixture. This may cause the test to hang.

[Cito]

I am now using the following decorator for async steps, which works well with pytest-asyncio:

def async_step(step):
    """Convert an async step function to a normal one."""

    signature = inspect.signature(step)
    parameters = list(signature.parameters.values())
    has_event_loop = any(parameter.name == "event_loop" for parameter in parameters)
    if not has_event_loop:
        parameters.append(
            inspect.Parameter("event_loop", inspect.Parameter.POSITIONAL_OR_KEYWORD)
        )
        step.__signature__ = signature.replace(parameters=parameters)

    @wraps(step)
    def run_step(*args, **kwargs):
        loop = kwargs["event_loop"] if has_event_loop else kwargs.pop("event_loop")
        return loop.run_until_complete(step(*args, **kwargs))

    return run_step

This can be applied in the same way:

@given("there is a cucumber", target_fixture="cucumber")
@async_step
async def there_is_a_cucumber():
    return 42

It works like this: The async_step decorator first checks whether the step function already requests the event_loop fixture of pytest-asyncio. If not, it requests it by adding it to the signature of the step function. The innner run_step function can now be sure that the event_loop is passed as a parameter, fetches it from there and runs the async function in that event loop.

[youtux]

Do you actually need all that signature machinery? Wouldn't this work as well?

import asyncio, functools

def async_to_sync(fn):
    """Convert async function to sync function."""

    @functools.wraps(fn)
    def wrapper(event_loop, *args, **kwargs):
        return event_loop.run_until_complete(fn(*args, **kwargs))

    return wrapper

[Cito]

Do you actually need all that signature machinery? Wouldn't this work as well?

See my other comment .

[youtux]

Note that my last example is different from what I originally suggested.

[Cito]

Note that my last example is different from what I originally suggested.

Sorry, yes, you're right. That simplified code should work, but does not cover the case that the original step function takes the event loop fixture as a parameter.

[youtux]

Ah yes you're right

[seifertm]

I come back to this issue from time to time, because there's a corresponding issue in pytest-asyncio.

My current understanding is that pytest-asyncio makes async step definitions hard to implement, because it assumes by default that each async test item runs in their own event loop. In pytest-bdd, however, you want to run multiple steps inside the same loop.

Up until pytest-asyncio v0.21, the only way to have an asyncio event loop scope other than function-scope is to reimplement the _eventloop fixture. This leads to a bunch of problems and the plan is to deprecated and remove this functionality.

As a replacement for event loop fixture overrides, pytest-asyncio will provide the _asyncio_eventloop mark (see https://github.com/pytest-dev/pytest-asyncio/pull/620). When the mark is added to a test class or to a module, pytest-asyncio will provide an asyncio event loop with the respective scope and run all tests under the mark in that scoped loop. It should also run any async fixture inside that same loop (although this needs more testing). Here's an example from the docs:

import asyncio

import pytest
import pytest_asyncio

@pytest.mark.asyncio_event_loop
class TestClassScopedLoop:
    loop: asyncio.AbstractEventLoop

    @pytest_asyncio.fixture
    async def my_fixture(self):
        TestClassScopedLoop.loop = asyncio.get_running_loop()

    @pytest.mark.asyncio
    async def test_runs_is_same_loop_as_fixture(self, my_fixture):
        assert asyncio.get_running_loop() is TestClassScopedLoop.loop

I'd be happy to provide a pre-release version to play around with this feature. Alternatively, you can just pip install from git, e.g. pip install git+https://github.com/pytest-dev/pytest-asyncio@106fa545a659a7e6a936b0f53d9d184287be8a13

Do you think this would help solve this issue?

[leferradkw]

I come back to this issue from time to time, because there's a corresponding issue in pytest-asyncio.

My current understanding is that pytest-asyncio makes async step definitions hard to implement, because it assumes by default that each async test item runs in their own event loop. In pytest-bdd, however, you want to run multiple steps inside the same loop.

Up until pytest-asyncio v0.21, the only way to have an asyncio event loop scope other than function-scope is to reimplement the _eventloop fixture. This leads to a bunch of problems and the plan is to deprecated and remove this functionality.

As a replacement for event loop fixture overrides, pytest-asyncio will provide the _asyncio_eventloop mark (see pytest-dev/pytest-asyncio#620). When the mark is added to a test class or to a module, pytest-asyncio will provide an asyncio event loop with the respective scope and run all tests under the mark in that scoped loop. It should also run any async fixture inside that same loop (although this needs more testing). Here's an example from the docs:

import asyncio

import pytest
import pytest_asyncio

@pytest.mark.asyncio_event_loop
class TestClassScopedLoop:
    loop: asyncio.AbstractEventLoop

    @pytest_asyncio.fixture
    async def my_fixture(self):
        TestClassScopedLoop.loop = asyncio.get_running_loop()

    @pytest.mark.asyncio
    async def test_runs_is_same_loop_as_fixture(self, my_fixture):
        assert asyncio.get_running_loop() is TestClassScopedLoop.loop

I'd be happy to provide a pre-release version to play around with this feature. Alternatively, you can just pip install from git, e.g. pip install git+https://github.com/pytest-dev/pytest-asyncio@106fa545a659a7e6a936b0f53d9d184287be8a13

Do you think this would help solve this issue?

Makes sense to me! The only caveat is forcing having tests inside a class to allow the mark @pytest.mark.asyncio_event_loop, which may not be natural to develop GWT tests with this library. Can you provide an example of the Gherkin file + corresponding async tests to understand the expected outcome?

[seifertm]

I imagine that pytest-bdd applies the mark under the hood via pytest.Item.add_marker. The pytest-bdd user shouldn't have to get in touch with the marker in my opinion.

That being said, the pytest-asyncio-0.22.0 release has been yanked. It's currently unclear if markers will be the way to go.

[ShedPlant]

As a much simpler workaround, you can (at least with pytest-bdd 7.3.0, pytest-asyncio 0.23.5) define in two steps:

@fixture
async def provisioned_thing(client, thing):
    await client.create(thing)
    return thing

@given("a thing")
def thing(provisioned_thing):
    return provisioned_thing