Nondeterministic behavior with conditional create

[splatch]

Hey, I wanted to have a conditional reference and conditional creation of resource. I've found it quite unstable giving on random factor errors: Invalid match URL "Patient?identifier=http://ehr.example.com|Patient/A9" - No resources match this search

When I change identifier and just try to execute transaction - all goes fine. I've tried with default value of reuse cache search results for millis as well as DaoConfig.setReuseCachedSearchResultsForMillis(null). Both cause non deterministic results. I was also switching order of resources - it doesn't matter - semi random distribution is still there.

Here is example request:

{
  "resourceType": "Bundle",
  "type": "transaction",
  "entry": [{
    "resource": {
      "resourceType": "Observation",
      "status": "final",
      "category": [{
        "coding": [{
          "system": "http://hl7.org/fhir/observation-category",
          "code": "vital-signs",
          "display": "Vital Signs",
          "userSelected": false
        }],
        "text": "Vital Signs"
      }],
      "code": {
        "coding": [{
          "system": "http://loinc.org",
          "code": "55284-4",
          "display": "Blood pressure systolic and diastolic",
          "userSelected": false
        }],
        "text": "Blood pressure systolic and diastolic"
      },
      "subject": {
        "reference": "Patient?identifier=http://ehr.example.com|Patient/A9"
      },
      "performer": {
        "reference": "Patient?identifier=http://ehr.example.com|Patient/A9"
      },
      "effectiveDateTime": "2009-09-09T00:00:00",
      "component": [{
        "code": {
          "coding": [{
            "system": "http://loinc.org",
            "code": "8480-6",
            "display": "Systolic blood pressure",
            "userSelected": false
          }],
          "text": "Systolic blood pressure"
        },
        "valueQuantity": {
          "value": 109,
          "unit": "mmHg",
          "system": "http://unitsofmeasure.org",
          "code": "mm[Hg]"
        }
      },{
        "code": {
          "coding": [{
            "system": "http://loinc.org",
            "code": "8462-4",
            "display": "Diastolic blood pressure",
            "userSelected": false
          }],
          "text": "Diastolic blood pressure"
        },
        "valueQuantity": {
          "value": 66,
          "unit": "mmHg",
          "system": "http://unitsofmeasure.org",
          "code": "mm[Hg]"
        }
      }],
      "comment": "BUNDLE testing."
    },
    "request": {
      "method": "POST",
      "url": "Observation"
    }
  },{
    "resource": {
      "resourceType": "Patient",
      "identifier": [{
        "use": "usual",
        "system": "http://ehr.example.com",
        "value": "Patient/A9"
      }],
      "active": true,
      "name": [{
        "use": "official",
        "family": "Chalmers",
        "given": [
          "Peter",
          "James"
        ]}
      ],
      "gender": "male",
      "birthDate": "1974-12-25"
    },
    "request": {
      "method": "POST",
      "url": "Patient",
      "ifNoneExist": "identifier=http://ehr.example.com|Patient/A9"
    }
  }]
}

[jamesagnew]

Hi @splatch,

I'm not sure I follow what you mean by "quite unstable giving on random factor errors".

Do you mean that when you post the Bundle above to an empty server, sometimes it succeeds and sometimes it fails? What is the error?

[splatch]

Hi James, Thank you for quick reply & sorry for lack of clearance. Yes - on empty server this request might fail on first, sometimes second attempt and succeed afterwards. When I lift up identifier I again have sometimes success, sometimes failure. Error message I get is: Invalid match URL "Patient?identifier=http://ehr.example.com|Patient/A13" - No resources match this search

[jamesagnew]

Ok interesting. FYI match URL processing like the the ones here bypass the query cache so that shouldn't be an issue.

I'll try running this same bundle and see if I get the same result. Is this happening on the latest release? (3.1.0)

[splatch]

Yes, it happens for me with 3.1.0, I was able to reproduce it using your test instance. It also fails under HAPI 2.4 (tested with hspc sandbox).

[jamesagnew]

Wait, looking at this more- I'm actually kind of surprised that this ever works. I woudn't expect the match URL to be able to find something that doesn't yet exist in the database.

Is there a reason you're not using a placeholder ID for the reference? E.g. something like this one: http://hapifhir.io/doc_rest_client_examples.html

[splatch]

I was digging around and found that

1. It is possible to use "search" in reference
2. It is possible to make operation conditional
3. Each aboce thing worked fine alone

Then I merged both approaches into one transaction boundary and it worked (with some error factor). Reason why I didn't use placeholder ids is quite simple - I didn't know about such possibility.

I think reason why this may work is hibernate and it's magic. Its either session cache or search which keep finding this resource. ca.uhn.fhir.jpa.dao.BaseHapiFhirDao#processMatchUrl sometimes returns identifiers, sometimes it does not.

[jamesagnew]

Hmm weird- I'm definitely unclear on how that could work.. Maybe if the resources were in the bundle in the opposite order. In general FHIR says that Bundle order shouldn't matter, but we do insert in the order that they appear in the bundle. But in this case we're adding something with a reference to something that comes after it.

I've added a unit test to verify that this fails (as would be my expectation).

Placeholders are definitely not the most obvious thing if you haven't used them before.. They're definitely the "FHIR way" to do what you want to do though.

[splatch]

I checked placeholder identifiers and this way transaction works as expected.

I thought this might be an ordering issue but after making patient at beginning of set - I had same error factor. The processMatchUrl method is called twice, once for "if-none-match" and second time for reference processing. I also think that transaction and database used behind may change behavior (I did run it against mysql) - depending on isolation level a read operations from inside of transaction might find a match. However, this is a case, as you said, only if patient is inserted before observation.

As my issue is solved I am fine to close this report. Thank you very much for support.

[jamesagnew]

No problem :)