search

medic / chis-interoperability

5 stars 1 forks source link

Configure CHT to send newly created patients to the OpenHIE ecosystem in FHIR format #1

Closed michaelkohn closed 3 years ago

michaelkohn commented 3 years ago

Our OpenHIE ecosystem consists of these components:

Our first step is to just have each CHIS be able to send a FHIR Patient resource to the interoperability layer and have the interoperability layer be able to send that patient resource to the HAPI FHIR server and see it saved.

We already have a CHT instance set up for this. An instance with the OpenHIE ecosystem (consisting of the OpenHIM and HAPI FHIR server) has already been set up and is currently being configured by ADP and ONA.

The two main things we'll need to do for the CHT are:

  1. Configure the CHT instance to be able to generate the FHIR patient resource that has been agreed upon. tl;dr:

    {
  "resourceType": "Patient",
  "gender": "male",
  "name": [
    {
      "use": "official",
      "family": "Jacobs",
      "given": [
        "Jeffrey"
      ]
    }
  ],
  "identifier": [
    {
      "use": "usual",
      "value": "cht/668011"
    }
  ],
  "birthDate": "1996-07-08",
  "telecom": [
    {
      "system": "phone",
      "value": "+14155551212",
      "use": "mobile"
    }
  ],
  "active": true
}

  2. The second step will be to have outbound send that FHIR resource to the appropriate OpenHIM endpoint (this hasn't been provided just yet).

Marc has already confirmed that the agreed FHIR patient resource can be generated from the CHT using outbound, and the mapping can be found here: cop_outbound_config.json.txt, so you should just need to take this mapping and install it on the CHT instance.

There's a chance that the OpenHIE ecosystem we are setting up will be ready to go by the time you start this, but if not, and as a test, you can also just configure outbound to send the payload to http://hapi.fhir.org/

Acceptance Criteria:

  1. Creating a new patient in the CHT triggers outbound push to send a payload to http://hapi.fhir.org/
  2. The details entered in #1 can be seen in http://hapi.fhir.org/

Once the internal OpenHIE infrastructure is configured, we will need to change the destination.

Below is an artists rendering of what the desired flow looks like as mapped over the OpenHIE architecture.

Screen Shot 2021-03-05 at 2 38 54 PM
mrjones-plip commented 3 years ago

This ticket is so pleasantly full of details and links, thanks @michaelkohn! I also see that the instance's credentials are in 1password :lock:. I'll poke around a bit on this next week, but likely will dive into this later in the week of the 15th so that I can focus on medic/cht-docs/issues/441.

We can chat more on our Monday call!

mrjones-plip commented 3 years ago

Helpful tip from QA about running dummy web server to help test outbound push.

mrjones-plip commented 3 years ago

@michaelkohn - we're lookin' good! I was able to:

  1. set up a dev CHT instance using development.md
  2. create an empty config and then initialize it medic-conf initialise-project-layout
  3. copy over the default app_settings.json from cht-core
  4. edit the app setting to have an outbound stanza
  5. push it with medic-conf --local compile-app-settings upload-app-settings
  6. Add/edit patient
  7. see the outbound calls work on dummy OpenHIM placeholder

Dummy OpenHIM on left, CHT outbound call on right looks like this:

@abbyad or @michaelkohn - I'm a little gun shy about stripping out all the app settings in the json file, but it seems like there's a lot of bloat in there. Thoughts on how to best proceed?

Next up - after we strip down the config file per above - acceptance criteria per the ticket body above:

  1. Creating a new patient in the CHT triggers outbound push to send a payload to http://hapi.fhir.org/
  2. The details entered in #1 can be seen in http://hapi.fhir.org/
mrjones-plip commented 3 years ago

Note to @mrjones-plip (self):

PHP simple server is just:

<?php
$save = '';
if(isset($_GET) && sizeof($_GET)>0) $save .= var_export($_GET,1);
if(isset($_POST) && sizeof($_POST)>0) $save .= var_export($_POST,1);
$data = json_decode(file_get_contents('php://input'), true);
if(isset($data) && sizeof($data) > 0) $save .= var_export($data,1);
error_log("{$save}");
print "Thanks! Got " . strlen($save) . "bytes";

And then this chumpy to see the output nicely formatted:

tail -f /var/log/apache2/error.log | sed "s/\\\n/\\n/g"
mrjones-plip commented 3 years ago

OK - decided to just go for it try against http://hapi.fhir.org/ now that I see it's a public test instance. FYI - the URL we need to use to POST outbound calls to is not https://hapi.fhir.org but https://hapi.fhir.org/baseR4/Patient instead.

I set this up on my dev instance by just changing the outbound URL here. The result can be seen on the hapi site here, but I'm not sure how long they store these test records.

Here's the debug log from my dev instances sentinel:

2021-03-09 16:18:45 DEBUG: About to send outbound request
2021-03-09 16:18:45 DEBUG: {
  "url": "https://hapi.fhir.org/baseR4/Patient",
  "body": {
    "resourceType": "Patient",
    "gender": "female",
    "name": [
      {
        "use": "official",
        "family": "jane doe",
        "given": [
          "jane"
        ]
      }
    ],
    "identifier": {
      "use": "usual",
      "value": "cht/038f01d2-d746-4247-b5f9-cf6c2307953a"
    },
    "birthDate": "1988-02-09",
    "telecom": [
      {
        "system": "phone",
        "value": "",
        "use": "mobile"
      }
    ],
    "active": true
  },
  "json": true,
  "timeout": 10000
}
2021-03-09 16:18:45 DEBUG: result from outbound request
2021-03-09 16:18:45 DEBUG: {
  "resourceType": "Patient",
  "id": "1937137",
  "meta": {
    "versionId": "1",
    "lastUpdated": "2021-03-10T00:18:45.225+00:00"
  },
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><div class=\"hapiHeaderText\">jane <b>JANE DOE </b></div><table class=\"hapiPropertyTable\"><tbody><tr><td>Identifier</td><td>cht/038f01d2-d746-4247-b5f9-cf6c2307953a</td></tr><tr><td>Date of birth</td><td><span>09 February 1988</span></td></tr></tbody></table></div>"
  },
  "identifier": [
    {
      "use": "usual",
      "value": "cht/038f01d2-d746-4247-b5f9-cf6c2307953a"
    }
  ],
  "active": true,
  "name": [
    {
      "use": "official",
      "family": "jane doe",
      "given": [
        "jane"
      ]
    }
  ],
  "telecom": [
    {
      "system": "phone",
      "use": "mobile"
    }
  ],
  "gender": "female",
  "birthDate": "1988-02-09"
}
2021-03-09 16:18:45 INFO: Pushed 038f01d2-d746-4247-b5f9-cf6c2307953a to FHIR
2021-03-09 16:18:45 DEBUG: finished transition mark_for_outbound doc 038f01d2-d746-4247-b5f9-cf6c2307953a is unchanged seq 138-g1AAAAIjeJzLYWBg4MhgTmHgzcvPy09JdcjLz8gvLskBCjMlMiTJ____PyuDOZE9FyjAnpZqmGxgmYiuGIf2JAUgmWSPakKipWlqqjmxJjiATIiHmsAGNsHcyMwg0SSNWBMSQCbUQ03gBptgYmFkZG5IrC_yWIAkQwOQAhoyH2SKANiUxBRLszQDY5JMWQAxZT_CN5YGSWmWppYkmXIAYsp9hFuSzM0NkoyTSDLlAcQUcLi4QkLW2NjYwhgjXLIAwYWoGg
2021-03-09 16:18:45 DEBUG: transition results: [true,null,false]

@michaelkohn - lemme know what you think next steps are! Likely pushing this to https://cht-cop-interop.dev.medicmobile.org/? Maybe after we get feedback on if the app config json is too verbose after being copied from cht-core?

michaelkohn commented 3 years ago

I'm a little gun shy about stripping out all the app settings in the json file, but it seems like there's a lot of bloat in there. Thoughts on how to best proceed?

If it would take more than 15 minutes to figure out what should be removed, removing it and testing it, I wouldn't recommend spending that time on it for this proof of concept.

lemme know what you think next steps are! Likely pushing this to https://cht-cop-interop.dev.medicmobile.org/?

Yep! That sounds right. And when that's done, it would probably be best if you collaborate directly with the CoP team to coordinate pointing to the CoP's OpenHIM endpoint.

abbyad commented 3 years ago

The app-settings is indeed large, but for the most part is necessary and won't cause you problems. I can walk you through it to see what can be safely cut down.

mrjones-plip commented 3 years ago

If it would take more than 15 minutes to figure out what should be removed, removing it and testing it, I wouldn't recommend spending that time on it for this proof of concept.

Cool - thanks @michaelkohn. I'll take this advice (to not spend more time on it) with @abbyad's offer to help (over the long run) and press ahead.

lemme know what you think next steps are! Likely pushing this to https://cht-cop-interop.dev.medicmobile.org/?

Yep! That sounds right. And when that's done, it would probably be best if you collaborate directly with the CoP team to coordinate pointing to the CoP's OpenHIM endpoint.

Cool, I'll get the test config set up on cht-cop-interop.dev.medicmobile.org later this week and then...I think the Slack invite you sent to join digitalsquare-public.slack.com will take care of the second part, yeah? w00t!

mrjones-plip commented 3 years ago

Per private slack thread, URL to use for this should be:

http://cop.app.medicmobile.org:5001/fhir/Patient

@nsikak-e-akpakpan - I see that I was POSTing to https://hapi.fhir.org/baseR4/Patient in my testing above. Does the cop need the baseR4 part of the URL or is that not needed? I'm guessing I'll find out as soon as I try, but I thought you might save me a test cycle or two ;)

As well, do I need to include the authentication with these POST calls you mention in the slack thread?

mrjones-plip commented 3 years ago

@f-odhiambo - I'm stuck on the API not authenticating my call. The username and password that @nsikak-e-akpakpan gave me in Slack (both test) don't seem to work. I've tested in the browser and with curl calls with either GET or POST. Here's a sample call that I would expect some json back saying something like malformed resource or something, but instead I get back a 401:

curl -v  -H "Content-Type: application/json" -d '{"resourceType":"Patient","gender":"female"}' http://test:test@cop.app.medicmobile.org:5001/open/Patient
*   Trying 52.56.228.185...
* TCP_NODELAY set
* Connected to cop.app.medicmobile.org (52.56.228.185) port 5001 (#0)
* Server auth using Basic with user 'test'
> POST /open/Patient HTTP/1.1
> Host: cop.app.medicmobile.org:5001
> Authorization: Basic dGVzdDp0ZXN0
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 44
> 
* upload completely sent off: 44 out of 44 bytes
< HTTP/1.1 401 Unauthorized
< Content-Type: application/octet-stream
< Content-Length: 44
* Authentication problem. Ignoring this.
< WWW-Authenticate: Basic
< Date: Tue, 23 Mar 2021 22:12:03 GMT
< Connection: keep-alive

On the server, running docker logs openhim-core --follow, I see it's saying it's not authorized to access any channels:

2021-03-23T21:58:58.370Z [worker1] info: No basic auth details supplied, trying next auth mechanism if any...
2021-03-23T21:58:58.371Z [worker1] info: Could NOT authenticate via TLS: undefined, trying next auth mechanism if any...
2021-03-23T21:58:58.384Z [worker1] info: No channel matched the request /fhir/Patient
2021-03-23T21:58:58.385Z [worker1] info: The request, '/fhir/Patient', is not authorised to access any channels.

The server also complains a lot about Could not find user by email root@openhim.org repeatedly in the logs (794 times and counting ; )

My next step is to try and log into the web GUI and read the docs to see how the API should be accessed/fix the test/test credentials etc. Please let me know if we think medic/chis-visualizations/issues/12 blocks this!

CC - @nsikak-e-akpakpan (but I know you're busy right now!)

nsikak-e-akpakpan commented 3 years ago

@mrjones-plip -- I have a few mins to look into the server.

mrjones-plip commented 3 years ago

I think this is a good CLI test to use to rule out any CHT issues: curl -sv -H "Content-Type: application/json" -d '{"resourceType":"Patient","gender":"female"}' http://URL-OF-INSTANCE

so right now we get a 401 on cop.app.medicmobile.org:5001, but this same request on hapi.fhir.org/ is happy hapi with the payload:

curl -sv -H "Content-Type: application/json" -d '{"resourceType":"Patient","gender":"female"}' http://hapi.fhir.org/baseR4/Patient 
*   Trying 35.229.94.143...
* TCP_NODELAY set
* Connected to hapi.fhir.org (35.229.94.143) port 80 (#0)
> POST /baseR4/Patient HTTP/1.1
> Host: hapi.fhir.org
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 44
> 
* upload completely sent off: 44 out of 44 bytes
< HTTP/1.1 201 Created
< Server: nginx/1.14.0 (Ubuntu)
< Date: Tue, 23 Mar 2021 22:16:34 GMT
< Content-Type: application/fhir+json;charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: HAPI FHIR 5.3.0 REST Server (FHIR Server; FHIR 4.0.1/R4)
< ETag: W/"1"
< X-Request-ID: dIjgub6L8JnNq5nX
< Content-Location: http://hapi.fhir.org/baseR4/Patient/1964183/_history/1
< Last-Modified: Tue, 23 Mar 2021 22:16:34 GMT
< Location: http://hapi.fhir.org/baseR4/Patient/1964183/_history/1
< 
{
  "resourceType": "Patient",
  "id": "1964183",
  "meta": {
    "versionId": "1",
    "lastUpdated": "2021-03-23T22:16:34.314+00:00"
  },
  "text": {
    "status": "generated",
    "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><table class=\"hapiPropertyTable\"><tbody/></table></div>"
  },
  "gender": "female"
* Connection #0 to host hapi.fhir.org left intact
}

@nsikak-e-akpakpan and I are in touch on Slack and he's currently doing some server side work to see about resolving the issue.

mrjones-plip commented 3 years ago

Much thanks to @nsikak-e-akpakpan's hard work , the OpenHIM server was was redeployed with just the minimal services. He then created a cht test user for the API. After committing this change to this repo to update the outbound base_url to point to http://cop.app.medicmobile.org:5001/fhir/Patient I then pushed the config to the server:

medic-conf --url=https://medic:REAL-PASSWORD-HERE@cht-cop-interop.dev.medicmobile.org/ compile-app-settings backup-app-settings upload-app-settings

Then accessing the CHT instance and creating a test patient, the corresponding patient is created in OpenHIM which is then federated back to FHIR.

Closing this ticket but please re-open if more work is needed.

CC @michaelkohn @f-odhiambo

mrjones-plip commented 3 years ago

Quick follow up to give some info on how to debug API calls. On the main cop server, I ran docker logs openhim-core --follow which allowed me to tail the server. Then running the curl command curl -sv -H "Content-Type: application/json" -d '{"resourceType":"Patient","gender":"female"}' http://test:test@cop.app.medicmobile.org:5001/open/Patient. You would see fail, per above. When things were fixed, you would then see it succeed like so:

2021-03-23T23:13:08.559Z [worker1] info: Client (Test Client) is Authenticated.
2021-03-23T23:13:08.567Z [worker1] info: The channel that matches the request /fhir/Patient is: FHIR Server
2021-03-23T23:13:08.568Z [worker1] info: The request, '/fhir/Patient' is authorised to access FHIR Server
2021-03-23T23:13:08.569Z [worker1] info: Storing request metadata for inbound transaction
2021-03-23T23:13:08.579Z [worker1] info: Routing http(s) request
2021-03-23T23:13:08.827Z [worker1] info: executing primary route : FHIR Server
2021-03-23T23:13:08.828Z [worker1] info: primary route completed
2021-03-23T23:13:08.828Z [worker1] info: All routes completed for transaction: 605a7604574a85001295e646
michaelkohn commented 3 years ago

@mrjones-plip I created this patient and expected to see it a few minutes later on the HAPI FHIR server but it never showed up.

I also tried creating this patient at a different level in the hierarchy and still never saw it show up.

I was going to check to see if i could see what was going on by looking at the OpenHIM server, but I don't have the password. I imagine it is in 1 Password but I don't think i have access to it if so. Can you have a quick look to see what's wrong and also let me know how to find the password to that server?

mrjones-plip commented 3 years ago

@michaelkohn - oh no! This is the same issue as before, so tracking this in medic/chis-visualizations/issues/12 which I've just re-opened.