wf2 - get all encounters via FHIR API?

aleksa-krolls commented 3 weeks ago

Background, context, and business value

Right now, in WF2 we...

get all Patients created/updated since last sync
for each Patient, get Encounters

However, we've learned that when Encounters are updated/created --> the related Patient record is not updated. Therefore, if only Encounters are modified... this workflow will not fetch these records. The workflow only fetches records if the Patient AND Encounter records are modified.

Request

Can you pls assist with some API discovery and analysis? I think we can get all Encounters modified in OMRS using the FHIR REST API?

See more on FHIR apis: https://www.hl7.org/fhir/http.html

This request seems to return Encounters in bulk: /openmrs/ws/fhir2/R4/Encounter

Can you see if you can this seems to return all records or if we need to add paging?
Does it look possible to add a filter parameter (e.g., meta.lastUpdated={cursor} to the API request to only get newly updated Encounters?
Does this response look similar to the OMRS /encounters endpoint? Or should we consider using this only to get a list of IDs for encounters last updated... and then use the regular OMRS /encounters api (as is) to get the rest of the obs data?

Input

Same OMRS credentials as #2

Toggl

MSF 2024

aleksa-krolls commented 2 weeks ago

@mtuchi see LucidChart "getEncounters" tab and use cursor 2024-09-29for testing https://lucid.app/lucidchart/38ae80f0-0650-4fea-9a11-dc1e19f9bd4f/edit?view_items=d1UdCvJE-99m&invitationId=inv_7c1bdd58-2ffe-4d84-b6fa-b8738126262c

mtuchi commented 2 weeks ago

@aleksa-krolls i am curios for this cursor 2024-09-29 how many encounters are you expecting ?

mtuchi commented 2 weeks ago

@aleksa-krolls I manage to fetch all patients with updated encounters using /ws/fhir2/R4/Encounter endpoint, See #42 But It require adaptor changes that need joe's input, See work in progress here https://github.com/OpenFn/adaptors/pull/765

aleksa-krolls commented 2 weeks ago

@mtuchi great that it sounds like you figured something out here, but please walk me through the proposed approach before spending time on any adaptor work. (That costs extra time/money, so I want to make sure we're aligned on the way forward before going and have considered using generic http before doing a bunch of new adaptor work.) I'll book some time for tomorrow --> so feel free to pause work here until we chat, and move onto the next issue.

mtuchi commented 2 weeks ago

API Discovery

We can get R4/Encounter since lastUpdated using a query like this _lastUpdated: 'ge2024-09-29'.
- _lastUpdated: Filters resources based on the last time they were updated. You can use operators like gt (greater than), lt (less than), ge (greater than or equal to), and le (less than or equal to).
  - gt2024-09-01T00:00:00Z filters encounters updated after September 1, 2024.
  - lt2024-10-01T00:00:00Z filters encounters updated before October 1, 2024.
We can use resource.subject.reference to extract the patient id. See sample payload here
Some encounter uuid returns 404

OpenMRS Discovery

API endpoint Currently all adaptor operation uses /ws/rest/v1/ api path, So we can't make a get request to /ws/fhir2/R4/Encounter without modifying the adaptor code.
API response The API response for /ws/fhir2/R4/Encounter has a different structure which can not be auto page with the current implementation of request helper. I have created a request helper that will work for ws/fhir2/R4/Encounter in https://github.com/OpenFn/adaptors/pull/765

aleksa-krolls commented 1 week ago

hey @mtuchi last week we discussed that I want to move forward with this... anything else blocking you here? Or can I move to the top of backlog Ready column?

mtuchi commented 1 week ago

@aleksa-krolls Joe asked me to look into OpenMRS Documentation. Also he asked me if we have more time to design the adaptor function, See his comment here

Unfortunately the Swagger documentation was not helpful. According to the instruction i should get the swagger documentation when accessing http://msf-ocg-openmrs3-dev.westeurope.cloudapp.azure.com/openmrs/module/fhir/rest/swagger.json, but i am 404

aleksa-krolls commented 1 week ago

hey @mtuchi let's please chat about this after standup tomorrow AM

aleksa-krolls commented 1 week ago

@mtuchi I also can't find the Swagger docs - doesn't look like they have this enabled in their deployment. I will need to ask the client.

For now, I would only worry about a generic FHIR GET - we might get any of these resource. Don't worry about POST, PUT, etc... we don't know enough yet about OMRS's fhir api, nor do we have budget to spend a lot of time on this. cc @josephjclark

mtuchi commented 1 week ago

EOD Update WIP https://github.com/OpenFn/adaptors/pull/765

I have update the function to fhir.get() and add unit test. Waiting for @josephjclark feedback

mtuchi commented 1 week ago

Hiya @josephjclark how do you feel about this implementation using http adaptor 👇🏽 ?

cursor("2024-09-29");

get(
  "/ws/fhir2/R4/Encounter",
  { query: { _count: 100, _lastUpdated: `ge${$.cursor}` }, parseAs: "json" },
  (state) => {
    const { link, total } = state.data;
    state.nextUrl = link
      .find((l) => l.relation === "next")
      ?.url.replace(/(_count=)\d+/, `$1${total}`);

    state.encounters = state.data;
    return state;
  }
);

fnIf(
  $.nextUrl,
  get($.nextUrl, { parseAs: "json" }, (state) => {
    delete state.encounters.link;
    state.encounters.entry.push(...state.data.entry);
    console.log(state.encounters.entry.length);
    return state;
  })
);

I know the work around for paging is not very nice but should get the job done. This will buy us time for a better implementation of OpenMRS FHIR

josephjclark commented 1 week ago

@mtuchi That looks fine to me, this sort of thing is exactly what http is for. Wasn't pagination part of the problem here?

Openmrs should how have the fhir functions in it so it's up to you which you want to use.

If you're not going to use the new fhir functions then we should probably raise a PR to remove them. We're sort of compounding bad decision on bad decision here...

mtuchi commented 1 week ago

@josephjclark by replacing the count value with total i am able to fetch the remaining records if any. So the magic is on this bit of code here

 const { link, total } = state.data;
 state.nextUrl = link
      .find((l) => l.relation === "next")
      ?.url.replace(/(_count=)\d+/, `$1${total}`);

If we find nextUrl we make another get request to fetch the remaining records.

I personally don't like this implementation, I think OpenMRS adaptor should handle pagination weather i am using a fhir module or openmrs as is. But i am happy to use that implementation if it will buy us time for a better adaptor design

mtuchi commented 1 week ago

Hiya @aleksa-krolls i have updated the implementation to use the http adaptor for the time being. Once this PR #42 pass code review you can merge and test on app. I have updated the spec.yaml file so it should sync to app without any issues

OpenFn / openfn-lime-pilot