[Bug] canonical search queries not working

[swithinfoote]

Describe the bug Unfortunately canonical search queries are not working. By definition a canonical reference is defined as either a canonical URL or fragment. When using this as a search parameter search is returning no results. We have found the way to the cause of this issue and believe that it is a core problem therefore we are unable to fix it locally. The best way to replicate this issue is to use the QuestionnaireResponse resource. This resource has a property questionnaire which is a canonical reference to the Questionnaire resource that the response relates to. The questionnaire property takes the form "questionnaire":" {{API_URL}}/Questionnaire/{{Questionnaire-Id}}" A search for QuestionnaireResponses that contain this property should look like GET {{API_URL}}/QuestionnaireResponse?questionnaire={{API_URL}}/Questionnaire/{{Questionnaire-Id}} The search above returns NO result even though the syntax is correct. It is important to note that this applies to ALL canonical fields that are FHIR searchable - another good example is instantiates-canonical which exists on a number of resources including CarePlan and ServiceRequest.

To Reproduce Steps to reproduce the behavior:

1. Create a QuestionnaireResponse resource with a questionnaire property as per above.
2. Search for QuestionnaireResponse resources using the query above
3. You will get an empty search set.

Expected behavior You should get a searchset result with the previously created QuestionnaireResponse resource.

Screenshots If applicable, add screenshots to help explain your problem.

Versions (please complete the following information):

"name": "fhir-works-on-aws-search-es", "version": "3.12.0", "description": "FHIR Works on AWS search Elasticsearch implementation",

Additional context We have been able to verify this by modifying the following https://github.com/awslabs/fhir-works-on-aws-search-es/blob/9a6b132f33b0944ff3379d33f1ff0d5b72cf9da2/src/QueryBuilder/typeQueries/referenceQuery.ts#L53

    if (searchParamName === 'questionnaire') {
        return { terms: { [`${compiled.path}${keywordSuffix}`]: references } };
    } else {
        return { terms: { [`${compiled.path}.reference${keywordSuffix}`]: references } };
    }

By adding the conditional logic the search returns as expected. The reason is that the existing logic returns .reference for the search query that is sent to OpenSearch. This does not exist on that document property as it is canonical. Without the above change the query that goes to OpenSearch is

{
    "from": 0,
    "size": 20,
    "track_total_hits": true,
    "body": {
        "query": {
            "bool": {
                "filter": [
                    {
                        "match": {
                            "documentStatus": "AVAILABLE"
                        }
                    },
                    {
                        "match": {
                            "_tenantId": "{{tenant}}"
                        }
                    }
                ],
                "must": [
                    {
                        "terms": {
                            "questionnaire.reference.keyword": [
                                "Questionnaire/b0c248e0-8f85-471b-913d-2f75444cfee1",
                                "{{API_URL}}/R4/Questionnaire/b0c248e0-8f85-471b-913d-2f75444cfee1"
                            ]
                        }
                    }
                ]
            }
        }
    },
    "index": "questionnaireresponse-alias-tenant-moh",
    "preference": "9ea0d2f29b639f253f53001f8453fc29f283fb39b6f2078aa03fdeb9cad24209"
}

With the above modification the search query for OpenSearch is:

{
    "from": 0,
    "size": 20,
    "track_total_hits": true,
    "body": {
        "query": {
            "bool": {
                "filter": [
                    {
                        "match": {
                            "documentStatus": "AVAILABLE"
                        }
                    },
                    {
                        "match": {
                            "_tenantId": "{{tennant-id}}"
                        }
                    }
                ],
                "must": [
                    {
                        "terms": {
                            "questionnaire.keyword": [
                                "Questionnaire/b0c248e0-8f85-471b-913d-2f75444cfee1",
                                "{{API_URL}}/Questionnaire/b0c248e0-8f85-471b-913d-2f75444cfee1"
                            ]
                        }
                    }
                ]
            }
        }
    },
    "index": "questionnaireresponse-alias-tenant-{{tennant-id}}",
    "preference": "9ea0d2f29b639f253f53001f8453fc29f283fb39b6f2078aa03fdeb9cad24209"
}

Please feel free to reach out if you would like me to run through this issue with you.

Please also note that this is an urgent issue for us as we have no other way to search for Patient responses to a particular Questionnaire.

Cheers.

[Bingjiling]

Hi @swithinfoote ,

Thanks for reaching out!

For the two OpenSearch query you shared before and after the change, the only difference I can see is API_URL and tenant_id fields replaced with place holders. Could you please verify if that's the change you meant to show?

Meanwhile, could you please provide the complete steps to reproduce this issue? Including the request bodies used for creating Questionnaire and QuestionnaireResponse resources.

Thanks, Yanyu

[swithinfoote]

Hi @Bingjiling - thanks for your response. As you can see from the code snippet the key is reference e.g. questionnaire.reference.keyword vs questionnaire.keyword

[swithinfoote]

These payloads will demonstrate the beavior Questionnaire

{
  "resourceType": "Questionnaire",
  "identifier": [
    {
      "use": "usual",
      "value": "TestQuestionnaireTemplate",
      "period": {
        "start": "2023-03-07"
      }
    }
  ],
  "version": "1.0.0",
  "name": "TestQuestionTemplate",
  "title": "Test Questionnaire",
  "status": "draft",
  "subjectType": [
    "Patient"
  ],
  "publisher": "Tester",
  "description": "A test questionnaire",
  "useContext": [
    {
      "code": {
        "system": "http://terminology.hl7.org/CodeSystem/usage-context-type",
        "code": "workflow",
        "display": "Workflow Setting"
      },
      "valueCodeableConcept": {
        "text": "Test Assessment"
      }
    }
  ],
  "purpose": "Test for FHIR Works issue https://github.com/awslabs/fhir-works-on-aws-deployment/issues/826",
  "code": [
    {
      "code": "TestQuestionnaireTemplate",
      "display": "Test Questionnaire"
    }
  ],
  "item": [
    {
      "extension": [
        {
          "url": "http://hl7.org/fhir/StructureDefinition/questionnaire-itemControl",
          "valueCodeableConcept": {
            "coding": [
              {
                "system": "http://hl7.org/fhir/questionnaire-item-control",
                "code": "drop-down",
                "display": "Drop down"
              }
            ],
            "text": "Drop down"
          }
        }
      ],
      "linkId": "TestQuestion1",
      "text": "Test question 1",
      "type": "choice",
      "required": true,
      "answerOption": [
        {
          "valueCoding": {
            "display": "Yes"
          }
        },
        {
          "valueCoding": {
            "display": "No"
          }
        }
      ]
    }
  ]
}

QuestionnaireResponse

{
  "resourceType": "QuestionnaireResponse",
  "questionnaire":"http://atesturl.org/testpath/test-questionnaire",
  "status": "completed",
  "authored": "2023-03-20T08:23:05.918Z",
  "item": [
    {
      "linkId": "TestQuestion1",
      "text": "Test question 1",
      "answer": [
        {
          "valueCoding": {
            "display": "Yes"
          }
        }
      ]
    }
  ]
}

Search for the created questionnaire using the questionnaire property as a parameter

GET https://hapi.fhir.org/baseR4/QuestionnaireResponse?questionnaire=http%3A%2F%2Fatesturl.org%2Ftestpath%2Ftest-questionnaire

Expected response is a searchset with 1 match item in the bundle

Actual response from FHIR Works an empty searchset.

[nguyen102]

I was able to reproduce the error. As you mentioned the QuestionnaireResponse is being categorized as a reference instead of a canonical data type. I will discuss with the team about the next steps and respond here once we have an update.

[swithinfoote]

Thanks @nguyen102 - also note that this issue affects all canonical fields. Eg instantiatesCanonical on a care plan or service request. Cheers.

[nguyen102]

Hi,

It looks like FWoA does not currently support reference search types of canonical resource type correctly. We will investigate on how we can differentiate resource type canonical from reference.

In the meantime one workaround I found is to change the canonical search types in compileSearchParameters.[fhirVersion].json file (link). For example in the case of the questionnaire field in QuestionnaireResponse I change the type from reference to string. Example shown below.

    {
        "name": "questionnaire",
        "url": "http://hl7.org/fhir/SearchParameter/QuestionnaireResponse-questionnaire",
        "type": "reference",
        "description": "The questionnaire the answers are provided for",
        "base": "QuestionnaireResponse",
        "target": [
            "Questionnaire"
        ],
        "compiled": [
            {
                "resourceType": "QuestionnaireResponse",
                "path": "questionnaire"
            }
        ]
    },

to

    {
        "name": "questionnaire",
        "url": "http://hl7.org/fhir/SearchParameter/QuestionnaireResponse-questionnaire",
        "type": "string",
        "description": "The questionnaire the answers are provided for",
        "base": "QuestionnaireResponse",
        "target": [
            "Questionnaire"
        ],
        "compiled": [
            {
                "resourceType": "QuestionnaireResponse",
                "path": "questionnaire"
            }
        ]
    },

Commit here

When I recompiled fhir-works-on-aws-deployment with the changes from fhir-works-on-aws-search-es, searching for questionnaire works. Can you try this workaround to see if it works for you? Does searching for canonical fields as a string meet the requirements for your search use cases? If yes, you can replace all the "canonical" types in the compileSearchParameters.[fhirVersion].json file with "string". You can find canonical types by downloading the JSON schema from here and searching for #/definitions/canonical".

-Tim

[nguyen102]

I downloaded the full FHIR spec from here and opened the search-parameters.json file. That's the definition we use for generating the compileSearchParameters. Unfortunately, that definition file only describe the search type as "reference", but does not provide context that it's a canonical resource element. Therefore, we would need to update the compiler to also read in fhir-schema.json. The fhir-schema.json file includes metadata that the field is canonical.

search-parameters.json Excerpt below

  {
    "fullUrl" : "http://hl7.org/fhir/SearchParameter/QuestionnaireResponse-questionnaire",
    "resource" : {
      "resourceType" : "SearchParameter",
      "id" : "QuestionnaireResponse-questionnaire",
      "extension" : [{
        "url" : "http://hl7.org/fhir/StructureDefinition/structuredefinition-standards-status",
        "valueCode" : "trial-use"
      }],
      "url" : "http://hl7.org/fhir/SearchParameter/QuestionnaireResponse-questionnaire",
      "version" : "4.3.0",
      "name" : "questionnaire",
      "status" : "draft",
      "experimental" : false,
      "date" : "2022-05-28T12:47:40+10:00",
      "publisher" : "Health Level Seven International (FHIR Infrastructure)",
      "contact" : [{
        "telecom" : [{
          "system" : "url",
          "value" : "http://hl7.org/fhir"
        }]
      },
      {
        "telecom" : [{
          "system" : "url",
          "value" : "http://www.hl7.org/Special/committees/fiwg/index.cfm"
        }]
      }],
      "description" : "The questionnaire the answers are provided for",
      "code" : "questionnaire",
      "base" : ["QuestionnaireResponse"],
      "type" : "reference",
      "expression" : "QuestionnaireResponse.questionnaire",
      "xpath" : "f:QuestionnaireResponse/f:questionnaire",
      "xpathUsage" : "normal",
      "target" : ["Questionnaire"]
    }
  },

fhir.schema.json excerpt below

          "questionnaire": {
          "description": "The Questionnaire that defines and organizes the questions for which answers are being provided.",
          "$ref": "#/definitions/canonical"
        },

[nisankep]

FHIR Works on AWS has been moved to maintenance mode. While in maintenance, we will not add any new features to this solution. All security issues should be reported directly to AWS Security at [aws-security@amazon.com] (mailto:security@amazon.com). If you are new to this solution, we advise you to explore using [HealthLake] (https://aws.amazon.com/healthlake), which is our managed service for building FHIR based transactional and analytics applications. You can get started by contacting your AWS Account team. If you are an existing customer of FHIR Works on AWS, and have additional questions or need immediate help, please reach out to fwoa-migration-support@amazon.com or contact your AWS Account team.