Implement test worker/test framework

[fabiocav]

Implement a test worker and test helpers for public consumption.

This should also be adopted by the worker tests in the project.

[fabiocav]

Work here is in progress, but likely to span across multiple sprints. We'll decompose this issue and assign more granular tasks.

[SeanFeldman]

@fabiocav, I'm assuming once some of that work can be shared, it will be. Looking forward to trying out what's being built.

[joshmouch]

Is there any update on when this would be available?

[davidstarkcab]

Any updates? :)

[newbazz]

do we have any framework to unit test out-of-process azure functions?

[Arash-Sabet]

@fabiocav Any updates or availability ETA on this feature especially for .NET 6.0?

[fabiocav]

@brettsam is actively investigating this and has made some progress, but there have been higher priority items taking precedence. We'll continue to use this issue to provide updates as we go

[andersson09]

In the meantime does this help?

//Arrange
var context = Substitute.For<FunctionContext>();
var request = Substitute.For<HttpRequestData>(context);
var response = Substitute.For<HttpResponseData>(context);

var serviceCollection = new ServiceCollection();
serviceCollection.AddSingleton(Options.Create(new WorkerOptions{Serializer = new JsonObjectSerializer()}));

var serviceProvider = serviceCollection.BuildServiceProvider();
context.InstanceServices.ReturnsForAnyArgs(serviceProvider);

request.Headers.ReturnsForAnyArgs(new HttpHeadersCollection());
response.Headers.ReturnsForAnyArgs(new HttpHeadersCollection());
response.Body.ReturnsForAnyArgs(new MemoryStream());
request.CreateResponse().ReturnsForAnyArgs(response);

var function = new Function();

//Act
var result = await function.Run(request);
result.Body.Position = 0;
var content = await JsonSerializer.DeserializeAsync<ErrorResponse>(result.Body);

//Assert
Assert.Equal(HttpStatusCode.OK, result.StatusCode);
Assert.Equal("Error", content?.Message);

[lohithgn]

Those who are searching for options - here is one more. I have created Test Doubles for the required classes to get Unit Testing working for a Isolated Process model function app. Here is the repo - https://github.com/lohithgn/az-fx-isolated-unittest.

Hope this helps.

[ayayalar]

In the meantime does this help?

//Arrange
var context = Substitute.For<FunctionContext>();
var request = Substitute.For<HttpRequestData>(context);
var response = Substitute.For<HttpResponseData>(context);

var serviceCollection = new ServiceCollection();
serviceCollection.AddSingleton(Options.Create(new WorkerOptions{Serializer = new JsonObjectSerializer()}));

var serviceProvider = serviceCollection.BuildServiceProvider();
context.InstanceServices.ReturnsForAnyArgs(serviceProvider);

request.Headers.ReturnsForAnyArgs(new HttpHeadersCollection());
response.Headers.ReturnsForAnyArgs(new HttpHeadersCollection());
response.Body.ReturnsForAnyArgs(new MemoryStream());
request.CreateResponse().ReturnsForAnyArgs(response);

var function = new Function();

//Act
var result = await function.Run(request);
result.Body.Position = 0;
var content = await JsonSerializer.DeserializeAsync<ErrorResponse>(result.Body);

//Assert
Assert.Equal(HttpStatusCode.OK, result.StatusCode);
Assert.Equal("Error", content?.Message);

For those of you using Moq, framework, only difference of what @andersson09 suggested is the following changes. Otherwise Body is always reinitialized and status code always returns 0.

// Status Code stubbing
mockHttpResponseData.SetupProperty(data => data.StatusCode);

// Body initialization 
var memoryStream = new MemoryStream();
mockHttpResponseData.Setup(data => data.Body).Returns(() => memoryStream);

[mkzer0]

What’s the latest on this issue? We’d really like to expand our api contract testing to isolated azure functions in dotnet. We can do this with TestSever for our dotnet core mvc apps. What’s the ETA?

[lohithgn]

@fabiocav would appreciate any update on this front. please let us know where this heading ?

[swiftbitdotco]

@fabiocav Is this being currently worked on?

[jimpudar]

Hi @fabiocav, there haven't been any updates on this feature for quite some time.

Is this something we can really expect to happen? If so, do we have any rough idea of the priority or an ETA?

If we can expect this sometime this year or even in 2023 Q1, I'd put less effort into making our own framework while writing tests for a new isolated functions service.

Thanks!

[josdeweger]

looking forward to an update on this also!

[kwacks]

Any updates on this? That the unit test story for isolated functions is half baked isn't called out anywhere in the documentation.

2 years since this issue was opened. Since in-process functions are not being iterated on, anyone who has a cache of large unit tests will run into this issue while migrating.

@fabiocav appreciate your work on this but is there any update/ETA on this?

[bhugot]

I think I m close to be able to do a PR that would allow integration testing using testserver handler for grpc. I have a working example on my fork. But some part are still hard coded.

[benjaminsampica]

This being half-baked is the primary reason we avoid using Azure Functions for things like APIs, instead favoring WebApis for its WebApplicationFactory harness. Please prioritize this 🙏🏻.

[mirekkukla]

Very much looking forward to this

[bhugot]

Hello as @fabiocav asked:

To enable integration testing using TestHost with grpc. Following things are needed:

1. Allow to pass a custom handler to grpc client so we can have communication without IO via TestClientHandler.

2. Implement the Host Server (2 solutions): a. Implement the logic of the server in Test library (the way I used in #1303) b. Having the test host directly provided by the https://github.com/Azure/azure-functions-host re-using maximum of code

a. Implement the logic of the server in Test Library need to develop alot of implementation from azure-function-host. Like handling the initilization of function at startup. A cool think is that we could specify only a substet of function if needed.

b. Create the test host in azure-functions-host would allow to be closer to the running version and also would allow to use all the triggers and test extensions but most of the code is in Host Application so it will need alot of refactoring to extract revelant code in another project to be packaged.

[Misiu]

I'm (again) new to Azure Functions (returning after a long break) and I've created a quite complex one lately (.NET7, Azure Functions V4, Isolated) and was trying to write integration tests. My function has middlewares and is doing a lot of external HTTP calls and data processing, so an easy way to set up an integration test with mocks would be great. Right now I use Postman to write tests (but there feel more like functional tests to me), but I must run my function locally to call my API instead of external API, etc. A clean and simple solution is needed and a lot of people are waiting for it.

[mohan810]

Hi @fabiocav ,

Can you please provide an update?

[mohan810]

Hello as @fabiocav asked:

To enable integration testing using TestHost with grpc. Following things are needed:

1. Allow to pass a custom handler to grpc client so we can have communication without IO via TestClientHandler.
2. Implement the Host Server (2 solutions): a. Implement the logic of the server in Test library (the way I used in Setup test framework #1303) b. Having the test host directly provided by the https://github.com/Azure/azure-functions-host re-using maximum of code

a. Implement the logic of the server in Test Library need to develop alot of implementation from azure-function-host. Like handling the initilization of function at startup. A cool think is that we could specify only a substet of function if needed.

b. Create the test host in azure-functions-host would allow to be closer to the running version and also would allow to use all the triggers and test extensions but most of the code is in Host Application so it will need alot of refactoring to extract revelant code in another project to be packaged.

Hi @bhugot ,

Can you please shar ean E2E example workin with 2.b?

[bhugot]

Hello @mohan810 it's. In the PR

[paul-datatech911]

This is really surprising & disappointing that we still don't have a cohesive test framework yet for a widely-used RTM product.

With the latest updates to the Isolated Worker libraries, I see a shift towards the ASPNet models with HttpRequestData -> HttpRequest and ConfigureFunctionWorkerDefaults -> ConfigureFunctionsWebApplication, but specifically middleware is excluded.

Is the plan to wire up a usable test configuration for Worker, or to keep moving Functions towards ASPNet? There would obviously be some big value in being able to reuse middleware from normal webpi. Any comments on this @fabiocav

[saalocin]

Any update on priority/progress test framework for isolated functions?

This will be blocking many people moving to PROD environment with the migration from inprogress to isolated.

[adrian-gheorghe]

Also very interested in this!

[alexjamesbrown]

As above, very interested in this.

[joshmouch]

Hello as @fabiocav asked:

To enable integration testing using TestHost with grpc. Following things are needed:

1. Allow to pass a custom handler to grpc client so we can have communication without IO via TestClientHandler.
2. Implement the Host Server (2 solutions): a. Implement the logic of the server in Test library (the way I used in Setup test framework #1303) b. Having the test host directly provided by the https://github.com/Azure/azure-functions-host re-using maximum of code

a. Implement the logic of the server in Test Library need to develop alot of implementation from azure-function-host. Like handling the initilization of function at startup. A cool think is that we could specify only a substet of function if needed.

b. Create the test host in azure-functions-host would allow to be closer to the running version and also would allow to use all the triggers and test extensions but most of the code is in Host Application so it will need alot of refactoring to extract revelant code in another project to be packaged.

Whichever option you choose, I like code re-use. There are a lot of "gotchas" that makes the MVC TestServer not usable in certain circumstances. For example, TestClient doesn't have the same stack of HttpHandlers as a non-test HttpClient. This means you can't test telemetry (AppInsights, OpenTelemetry), distributed transactions, and other things. I'd prefer if they'd used a regular HttpHandler and modified it as needed. I think another one is you can't inject HttpHandlers into DI to be used on HttpClient creation (can't remember if that's accurate). It's also harder to set up or modify the TestServer to the same extend as can be done with a regular IServer/IHost. For example, you have to have a Program.cs to use to set it up rather than being able to set it up in-line without referencing an external class.

At any rate: code reuse = ++++

[bhugot]

You are wrong on this as you can create the test handler only . And so add all the delegating handler you want. The only real problem I ever met was that the content stream was not the same for the test handler than Real http call. And didn t check if it was change in later version

[joshmouch]

You are wrong on this as you can create the test handler only . And so add all the delegating handler you want. The only real problem I ever met was that the content stream was not the same for the test handler than Real http call. And didn t check if it was change in later version

Hmm.. I guess I'll have to revisit. When I last looked, the HttpClient had a SocketHandler that got in the way, but maybe that can be bypassed and replaced with the DelegateHandler wrapped around a TestServerHandler? And going from the other direction, DiagnosticHandler wasn't public and couldn't be added to the TestHttpClient.

https://github.com/dotnet/aspnetcore/issues/24633

[Barsonax]

Iam quite stumped there's no easy way to do integration tests with azure functions like we can do with asp .net. For me this really feels azure functions is not mature yet. What surprises me even more that this issue is quite old already. Does this not have priority?

[jvmlet]

@Barsonax, if you read carefully, in one of the comments of this or other linked issues, @fabiocav mentioned that unit test support is not such important and AZ func team has tasks with higher priority to implement. I can't imagine nowadays developing code that you can't unitest. The official advice to mock everything and invoke your function via static method, or execute and debug your function locally, looks like neglection of delevopers intelligence to me. My advice - stay away from technologies you can't deploy and execute integration tests in-house (at least in simulation or dockerized mode ) The $$$ you save running your code in serverless environment costs you more when you get surprises in production, because of the bug/cases you couldn't test, causing business downtime.

[Barsonax]

@Barsonax, if you read carefully, in one of the comments of this or other linked issues, @fabiocav mentioned that unit test support is not such important and AZ func team has tasks with higher priority to implement.

Ah missed that, if that's really the case then that priority is wrong imho. But then again that's just my naive opinion.

I can't imagine nowadays developing code that you can't unitest. The official advice to mock everything and invoke your function via static method, or execute and debug your function locally, looks like neglection of delevopers intelligence to me.

This is what I was doing but yeah it annoys me it's so hard to test the whole app. There will always be gaps if you only do simple unit tests.

My advice - stay away from technologies you can't deploy and execute integration tests in-house (at least in simulation or dockerized mode ) The $$$ you save running your code in serverless environment costs you more when you get surprises in production, because of the bug/cases you couldn't tests, causing business downtime.

Totally agree, we are actually moving some functionality away from functions to a simple asp .net api because it's less complex. Some of the triggers are nice though but iam definitely keeping the functionality there to a minimum until this is fixed.

[verdantburrito]

Yeah, there's a lot to be desired where writing automated tests against Azure isolated functions is concerned. There's also the hard dependency on Newtonsoft.Json (azure-functions-openapi-extension#issue154), inaccessibility of ServiceBusReceivedMessage from middleware (azure-functions-dotnet-worker#issue1824), & inability to run Azure service bus locally to test ServiceBusTrigger function (azure-service-bus#issue223) that make testing more difficult than it should be. Some of these tickets have been around for years & seen very little activity, aside from end-user devs asking for updates.

If I had to do it all over again, I'd probably do what @Barsonax did & use ASP.NET Web API (which is robustly supported) instead of Azure functions for some of the work I'm doing.

[SeanFeldman]

inaccessibility of ServiceBusReceivedMessage from middleware (https://github.com/Azure/azure-functions-dotnet-worker/issues/1824)

That one is going to be resolved in the coming days as it's already working and just missing documentation.

inability to run Azure service bus locally to test ServiceBusTrigger function (https://github.com/Azure/azure-service-bus/issues/223#issuecomment-1741114770) that make testing more difficult than it should be

If you're testing a trigger and need to pass in a raw message (ServiceBusMessage), you can use a testing model factory provided by the ASB SDK to create one.

But in general, I agree. Having a foundational testing framework would be beneficial.

[verdantburrito]

@SeanFeldman Amazing! And thank you for providing these updates, regardless of whether or not the underlying issues are being worked on. Communication -- of awareness, intent, and/or progress -- goes a long way towards allaying frustration.

[mattchenderson]

This item should also cover the related work captured in #304

[dioum2touba]

In the meantime does this help?

//Arrange
var context = Substitute.For<FunctionContext>();
var request = Substitute.For<HttpRequestData>(context);
var response = Substitute.For<HttpResponseData>(context);

var serviceCollection = new ServiceCollection();
serviceCollection.AddSingleton(Options.Create(new WorkerOptions{Serializer = new JsonObjectSerializer()}));

var serviceProvider = serviceCollection.BuildServiceProvider();
context.InstanceServices.ReturnsForAnyArgs(serviceProvider);

request.Headers.ReturnsForAnyArgs(new HttpHeadersCollection());
response.Headers.ReturnsForAnyArgs(new HttpHeadersCollection());
response.Body.ReturnsForAnyArgs(new MemoryStream());
request.CreateResponse().ReturnsForAnyArgs(response);

var function = new Function();

//Act
var result = await function.Run(request);
result.Body.Position = 0;
var content = await JsonSerializer.DeserializeAsync<ErrorResponse>(result.Body);

//Assert
Assert.Equal(HttpStatusCode.OK, result.StatusCode);
Assert.Equal("Error", content?.Message);

For those of you using Moq, framework, only difference of what @andersson09 suggested is the following changes. Otherwise Body is always reinitialized and status code always returns 0.

// Status Code stubbing
mockHttpResponseData.SetupProperty(data => data.StatusCode);

// Body initialization 
var memoryStream = new MemoryStream();
mockHttpResponseData.Setup(data => data.Body).Returns(() => memoryStream);

For using Moq, I share with you how to implement this solution using Moq to test Azure Function Isolated .Net 8

public static HttpRequestData CreateMockHttpRequestData(string body, string? schema = null)
{
    var functionContext = new Mock<FunctionContext>();
    var requestData = new Mock<HttpRequestData>(functionContext.Object);
    var serviceCollection = new ServiceCollection();
    serviceCollection.AddSingleton(Options.Create(new WorkerOptions { Serializer = new JsonObjectSerializer() }));

    var serviceProvider = serviceCollection.BuildServiceProvider();
    functionContext.Setup(context => context.InstanceServices).Returns(serviceProvider);

    var bodyForHttpRequest = GetBodyForHttpRequest(body);
    requestData.Setup(context => context.Body).Returns(bodyForHttpRequest);

    var headersForHttpRequestData = new HttpHeadersCollection();
    if (!string.IsNullOrWhiteSpace(schema))
    {
        headersForHttpRequestData.Add("Authorization", $"{schema} edd2545es.ez5ez5454e.ezdsdsds");
    }

    requestData.Setup(context => context.Headers).Returns(headersForHttpRequestData);

    return requestData.Object;
}

private static MemoryStream GetBodyForHttpRequest(string body)
{
    var byteArray = Encoding.UTF8.GetBytes(body);
    var memoryStream = new MemoryStream(byteArray);
    memoryStream.Flush();
    memoryStream.Position = 0;

    return memoryStream;
}

[fabricciotc]

Any updates?

[jaliyaudagedara]

I came up with this until there is something like we have for ASP.NET Core applications.

Creating Integration Tests for Azure Functions

[tombiddulph]

I came across this project https://github.com/wigmorewelsh/FunctionTestHost, just given it a go with a .net 6 isolated function and it works nicely.

[VictorGazzinelli]

Any updates on this?

[matt-lethargic]

I might as well ask as it's been a couple of months...... any updates?

[joshmouch]

It seems that the newish .NET Aspire framework is a great fit for testing isolated Azure Functions. They're using the Visual Studio "IDE Execution" API to attach a debugger to user code that was started indirectly as sub-processes, spawned by the DCP "orchestration process".

https://github.com/dotnet/aspire/blob/main/docs/specs/IDE-execution.md

This sounds identical to Visual Studio starting an Azure Function Host which then spawns sub processes (user Function code) containing the user code.

[jvmlet]

This is the approach I've implemented in my company , but still lacks the ability to control the DI container (substitute services with mocks and set them up), but this is also solvable... I still can't understand how Azure dev team expects developers to test AZ functions??? Testability is one of the most important aspects when choosing the technology, customers just can't fall in the "serverless costs you less" trap. Without being able to unit/integration test - stay away.

[MattTravis]

So we saw the upcoming end of LTS for .NET 6 in November 2024 and thought, "Crikey! We'd best get a move on and migrate to .NET 8. Might as well jump on to Isolated Process while we're at it.". We initially tried out the ASP.NET Core integration route, maintaining all the familiar IActionResult and HttpRequest types and interfaces, but found that performance tanked instantly by a very noticeable amount (<30ms avg response times for simple HttpTrigger functions under .NET 6 In Process, jumping to >120ms avg response time under .NET 8 Isolated Process. We had expected some performance hit, but that felt like a lot). We then started trying to migrate to the in-built model using HttpResponseData and HttpRequestData and while performance improved, unit and integration testing were damn near impossible. A number of years-long threads of disappointment like this one and no visible process towards a solution is pretty frustrating.

[Barsonax]

I strongly think that http api's in Azure functions are always going to be second citizen to just asp .net. Might as well use asp .net then.

[ciarancolgan]

I, like the rest of the commenters here am pretty stunned that there is no MS solution to this yet. I don't think we need to rehash why a robust set of unit tests is fundamental to all good codebases, so this feature really shouldn't have been released without a clear unit testing approach. I was mostly able to work around the testability limitations of the FunctionContext using a combination of what @dioum2touba showed above using the Moq library plus a few other tweaks. For example, my big issue is with the IInvocationFeatures type of the Features property - I cannot Mock it as i'd want using e.g: invocationFeatures.Setup(x => x.Get<IFunctionBindingsFeature>()).Returns(...) as the type Microsoft.Azure.Functions.Worker.Context.Features.IFunctionBindingsFeature is internal! Gah! I cant add a Fake implementation of 'IInvocationFeatures' either for the same reason - I will not be allowed to override the 'Get' and return this type. So my solution (don't judge me, i've been looking at this for a day and it's starting to blur) to test a custom AuthenticationMiddleware class is along the lines of:

[Fact]
public async Task Invoke_ApiAuthenticationFailed_FunctionNotTriggered_UnauthorizedResponseReturned()
{
    // Arrange
    Mock<IApiAuthentication> apiAuthentication = new(); // Custom Auth service
    Mock<ILogger<AuthenticationMiddleware>> logger = new();

    Mock<IHttpRequestDataFeature> httpRequestDataFeature = new();
    Mock<IInvocationFeatures> invocationFeatures = new();

    var functionContext = MockFunctionContextHelpers.CreateContext(new Dictionary<string, object?>(), invocationFeatures.Object);

    invocationFeatures.Setup(x => x.Get<IHttpRequestDataFeature>()).Returns(httpRequestDataFeature.Object);

    var mockHttpRequestData = MockFunctionContextHelpers.CreateHttpRequestData("{}", "1234");

    httpRequestDataFeature
        .Setup(x => x.GetHttpRequestDataAsync(It.IsAny<FunctionContext>()))
        .ReturnsAsync(mockHttpRequestData);

    apiAuthentication.Setup(x => x.AuthoriseAsync(It.IsAny<HttpHeadersCollection>())).ReturnsAsync(new ApiAuthorisationResult("Stranger danger."));

    // Inline delegate, so we can set & check updated FunctionContext properties.
    var functionTriggered = false;
    async Task NoOpFunctionDelegate(FunctionContext context)
    {
        functionTriggered = true;

        // Do nothing
        await Task.Run(() => { });
    }

    try
    {
        // Act
        var sut = new AuthenticationMiddleware(apiAuthentication.Object, logger.Object);
        await sut.Invoke(functionContext, NoOpFunctionDelegate);
    }
    catch (Exception e)
    {
        // There is still no good Unit testing solution for Azure Functions. In this case we cannot use Moq to return a Setup for the 'Get' of the type:
        // 'IFunctionBindingsFeature' which needs to be in the IInvocationFeatures collection to be able to call 'GetInvocationResult' on the FunctionContext. We want to do something like:
        // invocationFeatures.Setup(x => x.Get<IFunctionBindingsFeature>()).Returns(...)
        // ... but we can't as the type 'Microsoft.Azure.Functions.Worker.Context.Features.IFunctionBindingsFeature' is internal. 
        // We cant add a Fake implementation of 'IInvocationFeatures' either for the same reason - we will not be allowed to override the 'Get<T>' and return this type.
        // Track either: https://github.com/Azure/azure-functions-dotnet-worker/issues/281 or https://github.com/Azure/azure-functions-dotnet-worker/issues/2263 for updates.
    }

    // Now we are past the error, we can access the httpResponseData.Body from the 'mockHttpRequestData' setup above.
    var updatedHttpResponseData = mockHttpRequestData.GetHttpResponseData;
    updatedHttpResponseData.Body.Position = 0;
    using var streamReader = new StreamReader(updatedHttpResponseData.Body);
    var httpResponseDataBody = await streamReader.ReadToEndAsync();

    // Assert
    updatedHttpResponseData.StatusCode.Should().Be(HttpStatusCode.Unauthorized);
    httpResponseDataBody.Should().Be("{\"Status\":\"Authorization failed\"}");

    functionTriggered.Should().BeFalse("Api Authentication failed, function not triggered.");

    // Assert the params passed to the mocked functions are as required
    apiAuthentication.Invocations[0].Arguments[0].Should().BeEquivalentTo(new HttpHeadersCollection
    {
        { "Content-Type", "application/json" },
        { "Authorization", "Bearer 1234" },
    });

    logger.Invocations[0].Arguments[0].Should().Be(LogLevel.Warning);
    logger.Invocations[0].Arguments[2].ToString().Should().Be("Stranger danger.");
}

Yes, I know.

The only other bit you'd be interested in here is the MockFunctionContextHelpers class which is:

public static class MockFunctionContextHelpers
{
    public static MockHttpRequestData CreateHttpRequestData(string? payload = null,
        string? token = null,
        string method = "GET")
    {
        var input = payload ?? string.Empty;
        var functionContext = CreateContext();
        var request = new MockHttpRequestData(functionContext, method: method,
            body: new MemoryStream(Encoding.UTF8.GetBytes(input)));

        request.Headers.Add("Content-Type", "application/json");

        if (token != null)
        {
            request.Headers.Add("Authorization", $"Bearer {token}");
        }

        return request;
    }

    public static FunctionContext CreateContext(Dictionary<string, object?> bindingData = null, IInvocationFeatures invocationFeatures = null)
    {
        var bindingContext = new MockBindingContext(new ReadOnlyDictionary<string, object?>(bindingData ?? new Dictionary<string, object?>()));
        var context = new MockFunctionContext(bindingContext, invocationFeatures);

        var services = new ServiceCollection();
        services.AddOptions();
        services.AddFunctionsWorkerCore();

        services.Configure<WorkerOptions>(c =>
        {
            c.Serializer = new JsonObjectSerializer();
        });

        context.InstanceServices = services.BuildServiceProvider();

        return context;
    }
}

• the Mock types in here are pretty much lifted from @lohithgn 's solution: https://github.com/lohithgn/az-fx-isolated-unittest

[bjose7]

MockBindingContext

How did you define MockBindingContext ?