Consider structuring outputType as a FHIR Parameters resource

[brynrhodes]

For tests that don't specify structured output, add an outputType of FHIRParameters and use that to specify the expected results

[brynrhodes]

Thinking a bit more about this, this test structure is model independent, and I don't know that we want to introduce a FHIR dependency by requiring the output to be structured as a FHIR resource. There is a fair bit of complexity in a model-independent representation, but I don't think we can justify making the TestSchema FHIR-specific. @JPercival @suyashkumar thoughts?

[suyashkumar]

I definitely see where you're coming from on being careful about data model specific deps in the test definitions themselves vs. the runner, and that makes sense.

I suppose while the FHIR Parameters is one engine independent representation of a CQL result, it is not necessarily data model independent. Can other data models (say, structured QDM types) be represented cleanly in the FHIR Parameters resource? (e.g. serialize them similar to tuples, but maybe with some special annotation)? I suppose the fact that this test runner uses FHIR Operations is a FHIR dependency in of itself, but is still separate from the actual test definitions themselves, which seems much more reasonable/tolerable.

In general, I think having an engine independent representation of CQL evaluation results (and inputs?) would be a really handy thing, not only for expressing the intended test result but also for general interoperability. For example, if we had a canonical JSON representation for every kind of CQL output, that could be handy for talking to different engines. I suppose this is what I was initially hoping the FHIR Parameters/CQL Operations would be. Not only would this enable a test runner that can interoperate with any engine, it would enable any other ecosystem component to interoperate with every engine! Plus, it may address the "eval both sides of the test assertion" quirk we were discussing (not a huge deal).

Perhaps we’re not quite there yet, or it's not an immediate priority--and I think thats okay. Overall, there may be more downsides to introducing the FHIR Parameters as-is in the tests than gains. However if we had a data-model and engine-independent representation of CQL results (e.g. some JSON representation) I think that could be a good fit (but now we run into having 3 output representations--native engine, FHIR Parameters, and this CQL json haha)! That may also be a bit complex as well.

Anyway, I think it's reasonable to keep things roughly as they are, and focus our engergies on test coverage and other aspects of the test runner

[brynrhodes]

Discussion on DQIC call, proposal is to:

• Define an engine-independent JSON serialization of CQL (CQL grammar for lists and tuples is almost JSON already)
• For tests that are more complex than just primitive (beyond the 800 or so that are currently passing), specify output using this serialization
• Update the runner to deserialize this JSON as the "expected" and transform the FHIR-based output of $cql to this same format and use that for the comparison

[EvanMachusakNCQA]

We have discussed at Cthon 36 the idea of using the ELM as a data model for the output, e.g. for Interval[1,2] we would want the ELM Interval literal data fields like: { low: { value: 1, type: "Literal", resultTypeName: "{model urn}Integer" }, high: ..., lowClosed: ..., highClosed: ... }

The purpose of typed literals is to disambiguate a JSON return value of 1 as either a long, decimal, or integer. It is critical that engines when expected to return a decimal actually do so, a fact which could be lost in serialization if not explicitly captured as a discrete field. The ELM already handles these details.

To accomplish this we should create a JSON schema file.

[EvanMachusakNCQA]

After a secondary conversation, we agreed to resolve the type problems with extensions on Parameter that the test runner will check.