Unit/integration testing durable functions

[paulschroeder-tomtom]

💡 Feature description We were using durable functions to create an API (what we call the orchestration API - oAPI). This one processes requests and (HTTP-) calls other (primitive - our terminology) APIs (pAPI). We have created a durable function (DF) app and registered several functions via decorators for the different routes. I.e.

import azure.durable_functions as dfapp = df.DFApp(http_auth_level=func.AuthLevel.ANONYMOUS)

app.register_blueprint(projects_orchestrators)
app.register_blueprint(landing_page_bp)# /projects

@app.route(route="projects")
@app.function_name(name="projects")
@app.durable_client_input(client_name="client")
async def projects(req: func.HttpRequest, context: func.Context, client) -> func.HttpResponse:
    ...
    instance_id = await client.start_new("list_projects_orchestrator", client_input=client_input)
    ...
    retrun client.create_check_status_response(req, instance_id)

@projects_orchestrators.orchestration_trigger(context_name="context")
def list_projects_orchestrator(context: df.DurableOrchestrationContext):
    ...
    # the real magic happens here
    ...

💭 Describe alternatives you've considered What we now would like to do now, is to (integration) test the whole setup and treat the DF app like a black box. So our test should look like so:

@pytest.mark.asyncio
async def test_projects():
    # mock pAPIs
    httpretty.register_uri(httpretty.GET, 'https://papi1.com/foo', body=json.dumps({'data': 123}))
    httpretty.register_uri(httpretty.GET, 'https://papi2.com/bar', body=json.dumps({'data': 456}))

    request = func.HttpRequest(...) 

    # pseudocode code
    from function_app import projects as function_under_test
    result = await function_under_test(req=request, context, client)
    # /pseudocode code 

    assert result == ...

We were already doing quite some tinkering to even get the equivalent to result = await function_under_test(req=request, context, client) working. But everything is quite fragile and is still failing because, as we guess, stuff that is supplied by the runtime is still missing.

We also would really like to avoid any shell command calls to bring up the runtime and prefer to have everything in python to easily mock external dependencies (we are also able to emulate the storage via azurite).

So, our burning questions are:

• is the way we are trying to go a dead end?
• how do we correctly call the functions like project from above?
• how do we correctly mock the inputs like context and client of projects?
• is it even possible to bring up the runtime purely in python/pytest?
• if so, how?

Thank you!

[davidmrdavid]

Hi @paulschroeder-tomtom, thanks for reaching out.

Yeah we've received a bit of feedback about the lack of clarity for testing, so we realize this is an area that needs improvement. I think we can use your project as a driving case to try to improve the experience here.

So to answer your question directly: I am personally not immediately sure that there's a "best practice"/ approved way to mock the DF Client and DF Context today (but there should be!), but I'm pretty certain that it is possible at the expense of it being hacky Are these the main objects you're struggling to mock? If so, I can try to put together the utilities that would allow you to do this and release them soon-ish.

In general, if you can put together a minimal app (emphasis on minimal, so that it's something I can run on my end) that you can share with me and comment in there the specific functionality you're and struggling trying to mock, that would help me identify the missing utilities (if any) and help provide guide the testing strategy.

[brandonwatts]

@davidmrdavid We are also struggling to really unittest our durable functions in v2. The decorators while really nice to work with, are a pain to unravel when testing. In activity functions we are testing them like ._function._func(<inputs) which isnt a huge deal but the orchestrators are almost impossible to test. Ill give an example. Say you wanted to write a unittest for the following orchestrator:

@bp.orchestration_trigger(context_name="context")
def orchestrator(context: df.DurableOrchestrationContext):
    x: int = context.get_input()
    y = yield context.call_activity(plusone, x)
    return y

@bp.activity_trigger(input_name="number")
def plusone(number: int) -> int:
    return number + 1

It isnt immediately apparent to me how you would do this. We copied over a contextbuilder class similar to how you guys did here but it ends up bringing with it almost the entire DF project. Its brittle and just overly complex. Im not sure what the right answer is here but I would love to have a simple way to test orchestrators and suborchestrators.

[paulschroeder-tomtom]

Hey @davidmrdavid

totally agree with @brandonwatts . I quickly coded up a minimal example:

import logging

import azure.durable_functions as df
import azure.functions as func

ips_orchestrators = df.Blueprint()
app = df.DFApp(http_auth_level=func.AuthLevel.ANONYMOUS)

# /ips
@app.route(route="ips/{version}")
@app.function_name(name="ips")
@app.durable_client_input(client_name="client")
async def ips(req: func.HttpRequest, context: func.Context, client) -> func.HttpResponse:
    """
    Could be more complex, so it needs to be tested. Kept short just for the sake of this example.
    NOTE:
        As @brandonwatts said, the decorators "eat" up the function (I guess durable_client_input is returning
        None instead of the original, other decorators also magle it from a regular Callable to DF) and so
        the ips() does not end up in the scope if this file after importing it.
    """
    # ... do something
    # ... do something with the context
    client_input = dict(
        des_version=req.route_params.get('version')
    )

    instance_id = await client.start_new('list_ips_orchestrator', client_input=client_input)
    # ... do something more
    return client.create_check_status_response(instance_id=instance_id, request=req)

# Construct and send primitive API calls to create project
@ips_orchestrators.orchestration_trigger(context_name="context")
def list_ips_orchestrator(context: df.DurableOrchestrationContext):
    """
    I know about context.task_all() but I wanted to keep it like that, because it reflects more our use case.
    Same here with the decorators
    """
    des_version = context.get_input()['des_version']

    ip_v4 = yield context.call_http(
        method='GET',
        uri='https://ipv4.icanhazip.com',
    )
    ip_v6 = yield context.call_http(
        method='GET',
        uri='https://ipv6.icanhazip.com',
    )

    return dict(
        des_verison=des_version,
        ip_v4=ip_v4['content'],
        ip_v6=ip_v6['content'],
    )

app.register_blueprint(ips_orchestrators)

[paulschroeder-tomtom]

It already 3 months ago that we opened up that issue. Is there any development on that?

[lilyjma]

Hi @paulschroeder-tomtom , @brandonwatts - thank you for using Durable Functions! I'm a PM working on DF and would love to learn about your experience using the product. You can share your feedback in this quick survey to help influence what the team works on next. If you're building intelligent apps, there's also an opportunity to participate in a compensated UX study. Thanks!

[paulschroeder-tomtom]

Hey @lilyjma to share my feedback also here: I am still waiting for member of the DF team to appropriately answer on that request right here as well as several other bugs/issues/question and feature requests.

I am quite annoyed by the behavior of MS. You are always asking for more information, a minimal example or (like here) a survey, without anything in return. No solution, no help. It really feels one sided! Improve on that.

[lilyjma]

HI @paulschroeder-tomtom - thanks for the feedback and apologies for the delayed response. My original intention was to get back to you and Brandon after we wrap up planning, because that way I can tell you whether this is something we'll work on next. (The survey was intended to gather data to help with work prioritization for the upcoming semester.) That said, I do acknowledge that the turnaround time wasn't great, so I understand your frustration here.

The good news is that helping customers easily write Python unit tests is something we'll prioritize in the next semester, which starts in April. I can't give you a definitive date now on when the improved experience will go out, since there's existing current semester work that needs wrap up, but we do see the feedback here and will address this ASAP when in progress work is done. I'll be sure to keep this thread updated.

I'll add @davidmrdavid to chime in with anything he'd like to add.

[cjsimm]

Any movement on this? Also searching for a reliable way to mock and run integration tests on a durable app.

[mark-walle]

Our team is also looking for a way to unit / integration test DFs. @lilyjma you were offering an "ASAP" update on progress. As we approach 5 months since that offer, and 10 months since this ticket opened: is there an update on the progress?

[LeonardHd]

Hi @mark-walle, @paulschroeder-tomtom,

I would like to chime in too. While I am not a working on the Durable Functions framework nor Azure Functions product group we are also using Azure Functions (and the Durable Functions Framework) in Python. @davidmrdavid and @lilyjma are, as far as I know, working on landing the right solution but some internal changes need to be done first - once they can I am sure are happy to share more details.

Unit testing

For the moment let me share our approach that tests the generator functions similar to what is already possible for the normal azure functions (so I expect that the future solution will behave like this).

Context: the root cause is that once you start decorating your function (e.g. def hello_orchestrator(context: DurableOrchestrationContext): with @hello_orchestrator_blueprint.orchestration_trigger(context_name="context")) the imported function won't return the original user function anymore but actually the durable function internal handle function.

To work around this the easiest way is to split your code with the actual decorated function like this:

# Orchestrator
import datetime
from azure.durable_functions import DurableOrchestrationContext, Blueprint

hello_orchestrator_blueprint = Blueprint()

@hello_orchestrator_blueprint.orchestration_trigger(context_name="context")
def hello_orchestrator(context: DurableOrchestrationContext):
    _orchestrator_function(context)

def _orchestrator_function(context: DurableOrchestrationContext):
    expiry = context.current_utc_datetime + datetime.timedelta(minutes=1)

    while context.current_utc_datetime < expiry:
        yield context.call_activity("hello_world_activity", context.instance_id)
        yield context.create_timer(context.current_utc_datetime + datetime.timedelta(seconds=5))
    yield context.call_activity("hello_world_activity", context.instance_id)

You can then easily unit test your functions using next and send operations (use orchestration.send when you doing something like value = yield context.call_activity).

An example test could be (I combined a few scenarios):

import datetime
from unittest.mock import Mock, PropertyMock
from azure.durable_functions import DurableOrchestrationContext
import pytest

from hello_orchestrator import _orchestrator_function

def test_hello_orchestrator_expires():

    # Arrange
    call_activity_mock = Mock()
    create_timer_mock = Mock()

    context_mock = Mock(spec=DurableOrchestrationContext)
    context_mock.instance_id = "instance_id"
    context_mock.call_activity = call_activity_mock
    context_mock.create_timer = create_timer_mock

    type(context_mock).current_utc_datetime = PropertyMock(
        side_effect=[
            datetime.datetime(2021, 1, 1, 0, 0, 0),
            datetime.datetime(2021, 1, 1, 0, 0, 1),
            datetime.datetime(2021, 1, 1, 0, 0, 2),
            datetime.datetime(2021, 1, 1, 0, 5, 0), # This is the expiry time
        ]
    )

    orchestration = _orchestrator_function(context_mock)

    # Act & Assert
    # first call to an activity
    next(orchestration)
    call_activity_mock.assert_called_once_with("hello_world_activity", context_mock.instance_id)

    # first call to create_timer with 5 seconds in the future
    next(orchestration)
    create_timer_mock.assert_called_once_with(datetime.datetime(2021, 1, 1, 0, 0, 7)) 

    # second call to the final activity (after the expiry time)
    next(orchestration)
    call_activity_mock.assert_called_with("hello_world_activity", context_mock.instance_id)

    # the orchestration should stop here
    with pytest.raises(StopIteration):
        next(orchestration)

Please let me know if you find this helpful! I am currently working on a sample repo to share a few different variations of testing the different durable function scenarios.

@davidmrdavid feel free to add any remarks/thoughts! I tried to summaries what we were discussing the other day.

Integration testing

For integration testing I strongly recommend using the func CLI (see Azure Functions Core Tools) because you actually want to test the whole stack of your application and some parts of the Durable Function framework rely on some dotnet (C#) implementation. You can write a small fixture that starts your function and then triggers the functions via http trigger etc. To isolate your applications external dependencies (because you cannot monkeypatch the python code in the python worker as it's owned by the Azure Function Web Host) you could write Fakes (much in the spirit of a Fake (see Test Double of Martin Fowler).If you like I can share more details on how we integration test Azure (Durable) Functions.

[jhswart]

Also want to bump this as it's a big issue in our current project. I see what @LeonardHd has done, and we'll probably go down that route temporarily to test, but definitely not something we'd like to make permanent in our codebase. It would be a massive help if there is some update on this. @lilyjma is there any update on progress on the above?

[hexeth]

Thank you for your work @LeonardHd. I'd love to see a GH Repo of examples

[LeonardHd]

@hexeth I provided a first sample on the unit tests in this repo LeonardHd/azure-durable-functions-python-testing-samples that builds on the testing approaches I have used in my work. I will try to add a few more samples in the coming days (I will be busy the coming weeks, so hope it's understandable that it might take some time). My personal goal is to eventually collect some guidance and best practices around it!

[lilyjma]

Thanks everyone for the comments! This issue hasn't fallen off our radar. We've discussed it today and found the way forward. But given how the technical details worked out, there needs to be at least two releases on the Python worker side, which would take a few months for the change to go out. Unfortunately, I can't provide an exact timeline now, but I'll track the progress closely and provide updates here when I have them.

In the meantime, please use the workaround LeonardHd posted above. This was something he and our team developed together.

(cc @davidmrdavid)

[hexeth]

@hexeth I provided a first sample on the unit tests in this repo LeonardHd/azure-durable-functions-python-testing-samples that builds on the testing approaches I have used in my work. I will try to add a few more samples in the coming days (I will be busy the coming weeks, so hope it's understandable that it might take some time). My personal goal is to eventually collect some guidance and best practices around it!

Thank you for your work! Very much appreciated!

Thanks everyone for the comments! This issue hasn't fallen off our radar. We've discussed it today and found the way forward. But given how the technical details worked out, there needs to be at least two releases on the Python worker side, which would take a few months for the change to go out. Unfortunately, I can't provide an exact timeline now, but I'll track the progress closely and provide updates here when I have them.

In the meantime, please use the workaround LeonardHd posted above. This was something he and our team developed together.

(cc @davidmrdavid)

Thank you @lilyjma. In relation to this subject, I found that unit testing functions with the @bp.durable_client_input decorator proved very difficult.

It seemed that importing the function that had the decorator caused the decorator to be instantiated and any attempts to mock the decorator failed as a result. I was able to work around this by using:

import routes.submit as uut

class TestSubmitFunction(unittest.IsolatedAsyncioTestCase):
    def setUp(self):
        def kill_patches():
            patch.stopall()
            reload(uut)
        self.addCleanup(kill_patches)

        patch('azure.durable_functions.decorators.durable_app.Blueprint.durable_client_input', lambda *x, **y: lambda f: f).start()
        reload(uut)

This would unload the decorator, and reload it as an item that is patched, and allow me to properly test the rest of the function. I'm very new to unit testing, so this approach could just be from my own ignorance, but I could not find many resources on the subject.

[davidmrdavid]

Thanks @hexeth - is there a reason why the approach provided by @LeonardHd would not work for your blueprint case as well? After all, the basic idea is not to import the top-level function, but instead of helper function that contains the implementation.

[hexeth]

Thanks @hexeth - is there a reason why the approach provided by @LeonardHd would not work for your blueprint case as well? After all, the basic idea is not to import the top-level function, but instead of helper function that contains the implementation.

My goal was to avoid making non-standard changes to my code just to make unit testing possible. I really appreciate the work @LeonardHd is doing, but have avoided implementing the orchestrator workaround at this time for the same reason. My orchestrator is relatively simple, so I'm comfortable enough leaving it without testing while all you people smarter than me figure out a standard unit testing implementation :)