Support for observation-based data extraction

[lamurian]

User story

As a researcher, I need a form data extraction feature so that I can directly query longitudinal data as a bundle of observation resources.

Feature specification

Is your feature request related to a problem? Please describe. A form data extraction feature will be useful, especially when handling self-report questionnaires.

Describe the solution you'd like I'm thinking about an observation-based data extraction.

Describe alternatives you've considered None at the moment.

Additional context

My team is currently developing a personal mental health record platform. This platform aims to bridge clinical and academic needs. Users can periodically complete questionnaires capturing their mental health and well-being on this platform. On the other hand, researchers can query and extract the anonymized responses. Since this app is a personal mental health record, it will be very nice to have the data structured as observation resources. I specifically need calculated scores to be extracted as an observation.

Example

1. Suppose that we have a personal health questionnaire (PHQ-9)
2. A user fills in this questionnaire
3. With calculatedExpression, we got the total score (coded as LOINC/SNOMED-CT term)
4. Extract the total score as an observation
5. Query the observation based on the coded term
6. Extract cohort and longitudinal data

What I found so far

I found that LHC Forms has implemented this feature. From their repository, I think these three functions are relevant to discuss:

1. convertLFormsToQRAndExtractFHIRData
2. _getExtractValue
3. _createObservation

I don't have any background in software engineering, so please pardon any lacking in my interpretation. This is my current understanding of how these functions work, kindly let me know if there's anything I miss:

1. Convert LForms Data to QuestionnaireResponse
   • Function: convertLFormsToQRAndExtractFHIRData
   • Take LForms data as an input
   • Convert LForms response into QuestionnaireResponse
2. Initialize data structure
   • Function: convertLFormsToQRAndExtractFHIRData
   • Store the QuestionnaireResponse in an array (rtn in line 46)
   • Prepare a reference string qrRef based on the QuestionnaireResponse ID, which will be used later to link observations to this questionnaire response
   • Loop through each item in LForms data
3. Determine if the item can be extracted
   • Function: _getExtractValue
   • Recurse from child item to parents and check if any of them includes FHIR extension indicating that its value should be extracted
   • If the check return true and the item is not null, create an Observation resource
   • If the item has any category extensions, add these to the observation's category
4. Generate Observation resource based on item data
   • Function: _createObservation
   • Identify the data type of the item (e.g., integer, date, coding) and prepare the corresponding observation value
   • Create the appropriate FHIR element based on data type (valueQuantity, valueDate, etc.) and set its value
   • If the item allows multiple values (e.g., coding with multiple codes), create separate Observation resources for each value
   • Create a new Observation resource with metadata such as status, code, and question text
   • Link the observation to the QuestionnaireResponse by setting various fields (e.g., basedOn, subject, derivedFrom) based on the QuestionnaireResponse
5. Add Observation resources to the results
   • Function: convertLFormsToQRAndExtractFHIRData
   • Add each Observation resource created to the rtn array (check step 2)
   • Return the array containing the QuestionnaireResponse and all related Observation resources

Objectives

I delegated this task to @ryuuzake, and we discussed it prior to submitting this issue. We wish to understand how AEHRC intends to implement observation-based form data extraction. I suppose the way LForms handles it is quite straightforward, but I'd love to know if there's any concern we need to address or anticipate.

[fongsean]

Hi @lamurian, thanks for the detailed write up.

It's an invaluable addition to the renderer in terms of adding coverage of SDC, and provides an additional extract option on top of the existing StructureMap extraction https://smartforms.csiro.au/docs/operations/extract.

It looks pretty straightforward to me as well. Looks like it can be implemented and exposed as a standalone function, similar to something like https://github.com/aehrc/smart-forms/blob/main/packages/smart-forms-renderer/src/utils/manageForm.ts#L101-L123.

This standalone function can be exposed by the renderer library, and consumed in the host app. i.e. import { removeEmptyAnswersFromResponse } from '@aehrc/smart-forms-renderer';

---

As for the implementation part - in this case it will look something like (Feel free to change it, this is just a rough guess): function extractObservationBased(questionnaire: Questionnaire, questionnaireResponse: QuestionnaireResponse): Observation[] I don't think you have to modify the QuestionnaireResponse, therefore the only output is the Observation array.

I imagine the structure of the recursive function will using the pattern of https://github.com/aehrc/smart-forms/blob/main/packages/smart-forms-renderer/src/utils/updateQr.ts#L33-L88, except you will not be updating the QR, and you will be pushing Observation(s) into the aggregated Observation array. There are other examples of this pattern in the codebase, but this is the cleanest one to use as a reference.

Dealing with this kind of recursion isn't easy, so please shout out if you need any help when working through this Riza!

[fongsean]

Hi @ryuuzake,

I just made a change on the dev branch that involves using the generic recursive pattern for reading stuff from the Q + QR.

See https://github.com/aehrc/smart-forms/pull/978.

• This contains the generic read function.
• This contains the callback function. In my case, it recursively creates a QuestionnaireItem[] containing item.initial values.
• This is the entry point of the generic read function.

Hope it might provide some guidance/inspiration.

[ryuuzake]

Alright @fongsean , I'll check on your suggestions. Thank you for your help.

I'm in the middle of implementing the extractObservationBased function. I hope I can PR in a day or two