Provide guidance on unit testing orchestrations

[anthonychu]

Because of the way Durable JS uses generators, it's not very easy to write tests. #49 has some examples here but they're hard to write.

Wonder if we should publish a library to make this easier. Here's an attempt of an API that works for different test frameworks: https://github.com/anthonychu/test-durable-js-testing/blob/master/__tests__/DurableFunctionsOrchestratorJS.tests.js

Thoughts @ConnorMcMahon @cgillum?

[cgillum]

It looks like you're using an SDK object to declaratively create an orchestration. Am I understanding correctly?

const orchestrator = func.createMockInstance();
orchestrator
    .addCallActivity(jest.fn().mockReturnValue('Hello Tokyo'))
    .addCallActivity(jest.fn().mockReturnValue('Hello Seattle'))
    .addCallActivity(jest.fn().mockReturnValue('Hello London'))
    .addWaitForExternalEvent(jest.fn().mockReturnValue({ useTimer: true }))
    .addCreateTimer(jest.fn());

I would much prefer a technique where users can run tests against their existing orchestrator code. Have you considered an approach where users can hijack our context methods, like callActivity to return custom data? That's closer to what we do in C# and seems to work reasonably well. In some ways, I feel like this should be even easier in JavaScript because method hijacking is a thing that JS lets you do natively.

[anthonychu]

It's testing the existing orchestrator function code here. There are parts of it that I don't like but wanted to explore how to configure what events to replay using a fluent API and then execute the replays and get access to the return value and use typical testing stuff like spies and mocks.

These lines in the tests replace the durable-functions sdk that the orchestrator imports with a library that enables the testing/mocking behavior:

const mockDurableFunctions = require('../mockDurableFunctions');
const func = require('../DurableFunctionsOrchestratorJS');
jest.mock('durable-functions', () => mockDurableFunctions);

Then we create an instance of the orchestrator that is mockable and exposes a fluent API to set up the history to replay. Each of the addCallActivity() injects a function that is called when each callActivity() is executed in the orchestrator. They can be plain functions or in this case we're using mocks that allow us to spy on them.

const orchestrator = func.createMockInstance();
orchestrator
    .addCallActivity(jest.fn().mockReturnValue('Hello Tokyo'))
    .addCallActivity(jest.fn().mockReturnValue('Hello Seattle'))
    .addCallActivity(jest.fn().mockReturnValue('Hello London'))
    .addWaitForExternalEvent(jest.fn().mockReturnValue({ useTimer: true }))
    .addCreateTimer(jest.fn());

Then we run the orchestrator, replaying the history we set up and returns the functions that were called and the result.

const {calls, result} = orchestrator.run();

At this point it's just regular JS testing, checking the results and spies.

const [tokyo, seattle, london, waitForExternalEvent] = calls;
expect(tokyo).toHaveBeenCalledWith('Hello', 'Tokyo');
expect(seattle).toHaveBeenCalledWith('Hello', 'Seattle');
expect(london).toHaveBeenCalledWith('Hello', 'London');
expect(waitForExternalEvent).toHaveBeenCalledWith('myevent');
expect(result).toEqual([ 'Hello Tokyo', 'Hello Seattle', 'Hello London' ]);

It should be flexible enough to work with most JS test frameworks and also allows testing of partial execution of an orchestrator.

[anthonychu]

Related: https://github.com/MicrosoftDocs/azure-docs/issues/29306

[antempus]

This would be extremely valuable for unit tests, I've been using an extremely naive way of unit testing the orchestrator as generators are a little more challenging to unit test with durable-functions.