Test case "Bad profile" for DocumentRecipient comprehensive test collection

[aschnab]

Hi,

I have been been using the FHIR Toolkit to test my implementation of an MHD Document Recipient/Document Responder, and I have noticed a strange test test result for the "Bad Profile" test case in the "MHDv410 DocumentRecipient comprehensive" test collection.

The test sends an ITI-65 request containing the erroneous Bundle profile declaration "http://ihe.net/fhir/StructureDefinition/IHE_MHD", and, by its decription, expects the MHD server to respond with an HTTP 400 response (which my MHD server does). Looking into the test's script, I can see that it expects the following set of assertions to fail (expectedFailAssertionIdList): 'RM410_4'|'CMPDR_3_1'|'TR_1'.

However, the test as a whole fails for me. Inspecting the details of the test results, I can see the test failure resulting from the step "PDB Validations." Here, the following three substeps are marked as failing:

1) "PDB Validations - Request message." - here, the assertion "Bundle.meta.profile: Minimal Metadata or Comprehensive Metadata." (Assertion ID: RM410_4) fails. Since this assertion's ID is contained in the above expectedFailAssertionIdList, I suppose that this assertion failure is expected and should not cause the test as a whole to fail. 2) "If Comprehensive Metadata." - here, the assertion "Transaction must be labeled Comprehensive Metadata." (Assertion ID: CMPMETA_3) fails. This assertion's ID is NOT contained in the above expectedFailAssertionIdList, so I suppose that this point is what causes the test as a whole to fail - correct? 3) "Transaction response." - here, the assertion "Transaction: http 200 response." (Assertion ID: TR_1) fails. Since this assertion's ID is contained in the above expectedFailAssertionIdList (and, going by the test's description, this assertion's failure should be a main point of this test), I suppose that this assertion failure is expected and should not cause the test as a whole to fail.

So, by the looks of it, the second of the three above mentioned assertion failures is causing the test to fail here. However, the test's setup description ("Bundle.meta.profile contains an invalid value") is in direct contradiction to what this assertion checks for (and neither is it even in the DocumentRecipient's power to change this profile declaration on the request side).

Am I correct in my conclusion that the above described test failure is just due to the setup of this specific test in the toolkit, and not indicative of any deficiencies in the system under test?

[skbhaskarla]

Correct. Also affected by the same problem is the Missing Profile test. Please see the commit 82a768c on a separate branch with the fixes. Thanks

[skbhaskarla]

In future, please post to the IHE MHD group first so that others might take notice of the problem as well.

[aschnab]

Thanks! The tests are running fine on my setup now (interestingly, the "Missing Profile" test case was working fine for me even before the fix, though).

In future, please post to the IHE MHD group first so that others might take notice of the problem as well.

Will do.

Regards, Andreas

[skbhaskarla]

Let me fully explain this bug fix and why the Missing Profile appeared to have passed on your side.

The Bad Profile negative test (and the Missing Profile test) originally had incorrect comprehensive DocumentReference metadata, it was missing the DocumentReference content attachment creation element. When you had reported the issue, only the Bad Profile had correct metadata but incorrect expected assertion fail Id, but the Missing Profile test had both incorrect metadata and incorrect expected assertion Id. You can see this in the referenced commit https://github.com/usnistgov/asbestos/commit/82a768c1da4d38d70aea236aa39a5ff846812d8b.

As a requirement to Comprehensive DocumentReference metadata,

XDS DocumentEntry creationTime is required in the Optionality Table: https://profiles.ihe.net/ITI/TF/Volume3/ch-4.3.html#4.3.1

FHIR mapping: https://profiles.ihe.net/ITI/MHD/StructureDefinition-IHE.MHD.Comprehensive.DocumentReference-mappings.html#mappings-for-xds-and-mhd-mapping-xds

The Toolkit tests incorrectly set the expected assertion fail to CMPDR_3_1 which was missing creation https://github.com/usnistgov/asbestos/blob/82a768c1da4d38d70aea236aa39a5ff846812d8b/asbestos-war/src/main/webapp/data/TestAssertions/assertions.json#L43

When you had run the Missing Profile test, the (incorrect) assertion test subject failed, and test passed (incorrect). This is what appeared to have happened with your Missing Profile test result.

Once the required creation element was fixed in test metadata through this bug fix, the correct error was triggered at CMPMETA_3. https://github.com/usnistgov/asbestos/blob/82a768c1da4d38d70aea236aa39a5ff846812d8b/asbestos-war/src/main/webapp/data/TestAssertions/assertions.json#L32

Sorry for the confusion.

[aschnab]

Ah, now I see: the Missing Profile test was testing something different than advertised, but it was doing so consistently. Thanks for the explanation!