In-Memory Integration Testing

jaredcnance commented 7 years ago

The problem

I would like to be able to run integration tests using the Azure Function runtime.

Workaround

I can use a FunctionRunner like this to fire up the runtime and then validate the results after some specified timeout.

There are several problems:

If I attempt to debug the test, I won't be able to set breakpoints within the actual function code since they are not in the same process. I have to run the test from the command line and add some while(Debugger.IsAttached == false) {/*...*/} hack.
I am required to set some arbitrary timeout and hope the function completes execution during that time
I cannot run a single function, all functions within the compiled function project will be started when the test host starts
requires func.exe to be installed on the testing server in a hardcoded location

Desired Solution

It would be great if we had a solution similar to AspNetCore that might look something like:

// arrange
var testHost = new TestHost(relativePathToCsproj);

// act
await testHost.RunFunctionAsync("MyFunction", maxInvocations: 1);

// assert...

A few things I would like to see included:

specify which function should be executed (and probably the binding as well)
specify the max number of times the function should be invoked before closing the host
a reasonable debugging story
assembly/type load errors should reproduce in tests. This probably means the test project should be executed by the function runtime

If the tools to do this already exist (within this project or any other), please let me know.

paulbatum commented 7 years ago

I would start by looking at our own e2e tests. For example we have this test fixture that should get you started.

If you end up building a test harness along the lines of what you outlined above that could be pretty useful in general, you should let us know.

Reopen this if you're still blocked.

jaredcnance commented 7 years ago

It looks like that fixture is dependent upon System.Web.Http, will that be changing with the movement to dotnet core?

paulbatum commented 7 years ago

Good question. I'm not sure what test fixture updates are planned, @fabiocav should be able to answer.

jaredcnance commented 7 years ago

I don't want to try and craft a solution based on volatile infrastructure, but this is something that is sorely needed, especially when the risk of runtime errors is so high because of https://github.com/Azure/azure-webjobs-sdk-script/issues/992

It looks like JobHost may be the thing I'm looking for in the future?

But then there is also this FunctionsLocalServer which seems to just start a new AspNetCore web host by spawning a new process to run the host via StartHostAction.

Can anyone provide me with the current design/direction so I don't do work that boxes me into a corner in future versions? Between this repository and the azure-functions-cli repo, I'm thoroughly confused as to how the runtime actually works.

jaredcnance commented 6 years ago

Can we re-open this issue? I'm not sure why it is closed. I don't think a proper solution has been provided nor do I have enough information to produce one.

charisk commented 6 years ago

I'm also facing the same challenge and agree with @jaredcnance - it would be great if there was some guidance here.

jeffhollan commented 6 years ago

Spent some time this weekend playing with this during some travel. I shared my project here. As @paulbatum suggested it follows the same patterns used by the azure functions host project. It's a proof of concept but does allow me to execute the Azure Functions Host inside the test suite and not as an external executable. Has some kinks (wasn't able to ever get the breakpoints inside the function to fire when debugging) but should enable you to verify assembly loading as part of a test

https://github.com/jeffhollan/functions-test-helper

s-krawczyk commented 5 years ago

+1

danhamlin commented 5 years ago

Curious if @fabiocav has anything he is able to share about a proper test fixture for function apps.

riezebosch commented 5 years ago

Checkout my solution: https://github.com/riezebosch/AzureFunctions.TestHelpers. It indeed uses the JobsHost from the SDK with some helper functions & classes aside to invoke http-triggered functions and wait for orchestration completion.

jeffhollan commented 5 years ago

@riezebosch this is awesome! So glad someone took some time to look into this as my sample is very stale at this point. Will check this out

matt-lethargic commented 3 years ago

3.5 years on, is there any official advice on how to achieve this?

fabiocav commented 3 years ago

@matt-lethargic there have been recommendations from @jeffhollan and @paulbatum on this issue. Aside from that, that has been no other work done.

Please do understand that, though the original question/request was made a while ago, there have been no commitments to bring anything beyond what we've been able to share. This is not something that have seen enough demand to be prioritized over other items with higher impact.

We decided to keep this open, as the idea is good and valid, but I want to make sure that wasn't sending the wrong message. In the meantime, I'll add a help wanted label to this as it would be a good opportunity for community engagement.

milad-ghafoori commented 2 years ago

I am looking for guidelines and tips to write in-memory integration tests for a set of V4 Azure Functions (.Net Core 6 & C#) in isolation mode (out-of-process). Is there any relevant content/sample/article available?

jvmlet commented 2 years ago

@fabiocav , please tag this as feature. Being able to run integration tests locally is the basic requirement for modern frameworks/infrastructures. Moreover, as a customer, I can say that testability plays the key role when choosing technology stack for solution.

the idea is good and valid

The idea of DI with test support is rather pretty old and globally adopted than good and valid, the official documentation with static function testing looks like disparage to the intelligence of Azure developers.

This is not something that have seen enough demand to be prioritized over other items with higher impact.

I think I'm not the only one who can't imagine writing code without being able to run unit/integration test locally with debugger support. I think no need to measure the demand for such capability

riezebosch commented 2 years ago

@milad-ghafoori what do you mean by out-of-process? If you look at my repository I've done some examples and included some helper methods but other than that it just uses the bits and pieces to set up a jobhost from the official SDK. For me that gives me just enough guarantee that my functions will eventually run appropriately in their designated runtime.

I've done an online session on how I do that: https://youtu.be/o1EqbeVF1EA?t=1501

Azure / azure-webjobs-sdk