Large FHIR service base URL files cannot be handled by Inferno Test Kit

[austeny]

Athenahealth has been using the test kit to validate our FHIR Service Base URLs Bundle however we've found that large bundles cause Inferno issues. Due to how the Inferno Test Framework calls its backend services and how the Service Base URL test kit passes around the bundle response as input and outputs to the tests. If the bundle is ~100mb, the payload size of the GET /api/test_sessions/<test_session_id>/results API can be ~1gb which either timeouts the frontend request or overwhelms the browser until it crashes.

This does just seem to be a limitation of the Inferno Test Framework as a whole and not just specific to this test kit but I am wondering if there are ways to change the test kit input/outputs so the entire bundle is not included in the /results API multiple times.

[arscan]

Hi @austeny -- thanks for letting us know about the size of your service base URLs, particularly about how these use inputs/outputs to pass data between tests (and back out the API mulitple times, in practice). That is problematic in this case. We split it up like that so that users could only run the second test if they didn't expose a URL, but that causes an issue for big service base url bundles.

@cooperthompson mentioned he also had issues with using large bundles, and I believe his solution was to manually trim down the size of a bundle to use just for testing. Obviously that wouldn't check every entry, but I don't think you really need to check every entry to have high confidence that your implementation is broadly correct.

We could probably add a test that just grabs the URL and validates everything within a single test, and as long as we are careful about errors/messages generated perhaps that would be ok, as we separate the request/response payload into a separate API request.

We also have a broad feature we are looking at to solve this more generically, though that would be some ways off.

[austeny]

@arscan, Yeah totally understand. I think the ability to paste a bundle to validate against is very useful and I had made use of that early on.

Yeah when testing with a smaller bundle which contains a subset of data the Inferno test kit passes and performs without a problem. I guess my concern would be about the certification process and if testing against a subset would be sufficient. Yeah happy to hear that you guys are already thinking about optimizations.

[arscan]

Thanks for bringing this up @austeny. Disclaimer: I do not speak for ONC, but I try to be helpful. Asking for clarification directly via ONC's portal is the only way to get official answers.

The Service Base URL requirement falls under Conditions and Maintenance of Certification which is a bit different than certification to the (g)(10) criteria. There is an attestation relating to the publication in this standard format, but I don't see a specific test method, thus I do not think there is a technical requirement to pass these tests if you believe your implementation to be correct. I'm asking ONC for language we can include in the Service Base URL Test Kit to explain this.

Again, I do not speak for ONC here.

Hopefully we can address this issues though, I'd like these to be as helpful as possible, and I think the most helpful thing would be to 'pass' any data set that appears to meet the requirement (even if we don't specifically check EVERYTHING, like if the production bundle provides thousands of FHIR endpoints, due to test design constraints).

[austeny]

@arscan thanks for all the information and the disclaimer! Yeah I agree that since the goal of this test kit is to validate formatting of the data and not actual data itself then a passing test with a subset of the data should suffice. I just don't want to assume that encase we do need to test on the entire data set.

If there is anywhere I can try to ask some questions to ONC to throw a bit more weight behind the request for clarification I'm happy to help.

[cooperthompson]

I haven't gotten to testing our full bundle yet, but I'm kinda expecting to run into size issues as well. I probably should have brought this up during the Argo Brands project so we could have something like a manifest of Brands bundle files or something. It might be worth getting some agreement in the industry that publishing a set of Brands files is an ok practice.

[austeny]

@cooperthompson, happy to hear others are also concerned that they may run into this issue as well. I know that ONC did comment on the support of large bundles for FHIR Service Base URLs in HTI-1 Final Rule comments (https://www.federalregister.gov/d/2023-28857/p-1189). It pretty much reads that we know its large but expect developers to have the means to deal with it.

Maybe with HTI-2 in proposal there could be the ability to comment and request changes.