Allow recording backend responses to fixtures

I'm not sure if this would fit the scope of the CustomDataProvider, so I'll leave it open if we implement it there or somewhere else. But it seems nice to be able to record live backend responses to fixtures, for use in an app's test suite.

Currently the CustomDataProvider allows bypassing the RestAPILink, so that you can test without having to mock the app-runtime hooks (in an app's tests) that make network requests:

https://github.com/dhis2/app-runtime/blob/e9b73cac0afcb292454e8b50e5e4812d442f8950/services/data/src/__tests__/integration.test.tsx#L27

I created a small POC that uses nock to do something similar (because I wasn't aware of the CustomDataProvider). The CustomDataProvider seems like a similar solution, but one thing that nock has that would be nice to have in the CustomDataProvider as well, is fixture recording and playback with nock.back.

@amcgee and I were talking about it a bit, and it seems like we might be able to implement that with some additions to the CustomDataProvider (or another custom provider). The goal being to allow for fixture recording and playback.

Seems to me that that could allow for more realistic tests. Maybe not appropriate everywhere, but definitely a nice option to have in my opinion.

Discussed on slack here: https://dhis2.slack.com/archives/CLJSVA1RV/p1602062142003100

These libs might be useful as well:

So I took a closer look, and I wonder if in the slack thread @amcgee and I were misunderstanding eachother. So to clarify what the POC is doing:

It uses an express style syntax of matching requests to responses. So the calls to nock are defining interceptors, which intercept requests at the node http library, and reply with the specified response. For the purpose of the tests fetch has been polyfilled with node's http.
No calls are going through to an actual backend in this POC, it's all handled locally
The url used in the tests is just something random I chose. I can imagine that we'd use a standardised db and a fixed dhis2 version to record proper responses from. Point is, that can easily be changed to something more appropriate.

The downsides I see to the test style of the POC:

It doesn't test the component embedded in the entire app-runtime. There might be unforeseen interactions.
It mocks fetch to intercept requests, so there might be discrepancies at that level.
The backend base url and api version are set in the test, and might be different from what the app will use.

I feel that these downsides are acceptable though. For an an integration test like this I'm not expecting a 100% realism, and I feel it's still an improvement in realism over the test style that this component had before (mocking the hook), since we're mocking closer to the actual network traffic.

Now I'm not sure if I misunderstood @amcgee's feedback, or if my explanation of the POC was unclear. But the above are the only clear downsides I see (let me know if I'm missing anything Austin).

I took a look at the CustomDataProvider as an alternative. The mocking there is a little higher up than with the POC. It feels like it's in between mocking the hook and the POC in its level of realism. Arguments can be made for all I guess (though mocking the hook seems less useful when we have the CustomDataProvider). Not all tests need the added realism of the nock-style tests, so then it's easier to just use the CustomDataProvider.

But for those tests where it would be appropriate (recording and playback of responses), it seems most practical to me with a combination of the POC and a custom data provider. The one thing that I think is missing from the app-runtime's side to make it work is a TestProvider that allows injecting http basic auth into the request. So say:

            <TestDataProvider
                baseUrl="https://some.server/"
                apiVersion={32}
                authentication={/* or something more generic and high-level */}
            >

That should allow the tests to authenticate with a backend, record responses and use that as fixtures (unfortunately nock can't intercept and modify a request before it goes out). Code could stay quite similar to the POC, and hopefully it wouldn't be too complicated to add a provider like the above one. I think that'd be the most practical, simple way to do it.

dhis2 / notes

Allow recording backend responses to fixtures #142