awslabs / fhir-works-on-aws-deployment

A serverless implementation of the FHIR standard that enables users to focus more on their business needs/uniqueness rather than the FHIR specification
Apache License 2.0
298 stars 160 forks source link

Clinical Data #803

Closed youbenuse closed 1 year ago

youbenuse commented 1 year ago

Hello, I implemented CCDA[http://hl7.org/fhir/us/ccda/history.html] IGs according to the IG implementation guide. And want to check clinical document formated data. I already created Patient, Encounter, Location, Observation using FHIR-UI repository after deployment FHIR server on the AWS. But I can't get any clinical document data.

Would you like to help me what do I need to do for this? Even I can't find any difference after I implement CDEX and CCDA IG.

Hope to get your kind help. Best Regards.

Bingjiling commented 1 year ago

Hi @youbenuse ,

Thanks for reaching out!

Could you please provide more details in how to reproduce the scenario? Including:

  1. The version of IGs added
  2. The full requests sent to create the FHIR resources
  3. The expected behavior of getting clinical document data

Thanks, Yanyu

youbenuse commented 1 year ago

Hi, @Bingjiling Thanks for your kind reply... Screenshot 2023-01-27 at 16 33 20

hl7.fhir.us.davinci-cdex: v2.0.0 hl7.fhir.r4.core: v4.0.1 hl7.fhir.us.davinci-hrex: v0.2.0 hl7.fhir.us.ccda: v.1.1.0

  1. As you can see, I added several IGs and deployed compiledIGs to AWS using CDK.
  2. And regarding all of Patient, Observation, Encounter, so on, I created all of temp data using FHIR-UI(Updated by my self for all of resource templates) but I'm not sure how implemented IGs working for Clinical Data and even not sure how can I get Clinical Document. This is my created Patient data template.
    
    "photo": [
        {
            "data": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAApgAAAKYB3X3/OAAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAANCSURBVEiJtZZPbBtFFMZ/M7ubXdtdb1xSFyeilBapySVU8h8OoFaooFSqiihIVIpQBKci6KEg9Q6H9kovIHoCIVQJJCKE1ENFjnAgcaSGC6rEnxBwA04Tx43t2FnvDAfjkNibxgHxnWb2e/u992bee7tCa00YFsffekFY+nUzFtjW0LrvjRXrCDIAaPLlW0nHL0SsZtVoaF98mLrx3pdhOqLtYPHChahZcYYO7KvPFxvRl5XPp1sN3adWiD1ZAqD6XYK1b/dvE5IWryTt2udLFedwc1+9kLp+vbbpoDh+6TklxBeAi9TL0taeWpdmZzQDry0AcO+jQ12RyohqqoYoo8RDwJrU+qXkjWtfi8Xxt58BdQuwQs9qC/afLwCw8tnQbqYAPsgxE1S6F3EAIXux2oQFKm0ihMsOF71dHYx+f3NND68ghCu1YIoePPQN1pGRABkJ6Bus96CutRZMydTl+TvuiRW1m3n0eDl0vRPcEysqdXn+jsQPsrHMquGeXEaY4Yk4wxWcY5V/9scqOMOVUFthatyTy8QyqwZ+kDURKoMWxNKr2EeqVKcTNOajqKoBgOE28U4tdQl5p5bwCw7BWquaZSzAPlwjlithJtp3pTImSqQRrb2Z8PHGigD4RZuNX6JYj6wj7O4TFLbCO/Mn/m8R+h6rYSUb3ekokRY6f/YukArN979jcW+V/S8g0eT/N3VN3kTqWbQ428m9/8k0P/1aIhF36PccEl6EhOcAUCrXKZXXWS3XKd2vc/TRBG9O5ELC17MmWubD2nKhUKZa26Ba2+D3P+4/MNCFwg59oWVeYhkzgN/JDR8deKBoD7Y+ljEjGZ0sosXVTvbc6RHirr2reNy1OXd6pJsQ+gqjk8VWFYmHrwBzW/n+uMPFiRwHB2I7ih8ciHFxIkd/3Omk5tCDV1t+2nNu5sxxpDFNx+huNhVT3/zMDz8usXC3ddaHBj1GHj/As08fwTS7Kt1HBTmyN29vdwAw+/wbwLVOJ3uAD1wi/dUH7Qei66PfyuRj4Ik9is+hglfbkbfR3cnZm7chlUWLdwmprtCohX4HUtlOcQjLYCu+fzGJH2QRKvP3UNz8bWk1qMxjGTOMThZ3kvgLI5AzFfo379UAAAAASUVORK5CYII=",
            "title": "Avatar",
            "contentType": "image/png",
            "creation": "2022-01-10"
        }
    ],
    "birthDate": "2000-10-10",
    "meta": {
        "lastUpdated": "2023-01-13T03:42:16.926Z",
        "versionId": "1"
    },
    "managingOrganization": {
        "reference": "Organization/2.16.840.1.113883.19.5",
        "display": "Good Health Clinic"
    },
    "text": {
        "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><p>John Smith</p></div>",
        "status": "generated"
    },
    "address": [
        {
            "text": "Calrifonia 1",
            "state": "Los Angels",
            "type": "both",
            "city": "Industry",
            "use": "home",
            "postalCode": "90035"
        }
    ],
    "name": [
        {
            "family": "Test",
            "given": [
                "Patient 2"
            ]
        }
    ],
    "gender": "female",
    "active": true,
    "resourceType": "Patient",
    "generalPractitioner": [
        {
            "reference": "Practitioner/1",
            "type": "Practitioner",
            "identifier": {
                "value": "b27daf48-0220-4db1-9506-b2b90561152e",
                "use": "usual"
            }
        }
    ],
    "id": "9338c35a-a4f1-4146-991b-6346ac1f49ae",
    "communication": [
        {
            "language": [
                {
                    "coding": [
                        {
                            "system": "http://snomed.info/sct",
                            "code": "260385009",
                            "display": "Negative"
                        },
                        {
                            "system": "https://acme.lab/resultcodes",
                            "code": "NEG",
                            "display": "Negative"
                        }
                    ],
                    "text": "Negative for Chlamydia Trachomatis rRNA"
                }
            ],
            "preferred": true
        }
    ]
    }```
  3. I'm new in this FHIR so really not sure how can I get clinical data from the current AWS FHIR server. Due to my knowledge, CCDA is working to convert xml to FHIR format, But pretty not sure about CDEX either. So hope you guide me what I can I do for this...

Thank you so much.

Bingjiling commented 1 year ago

Hi @youbenuse,

Thanks for the details!

I have not used CCDA or CDEX IGs before. But I can point you to documentations on FHIR CRUD operation and how FWoA works with binary files. In FWoA binary resources are stored in S3 and structured data in DynamoDB. Based on the information we see, you are storing both binary and structured data. Here're some documentation that would be of help:

FHIR operations: https://www.hl7.org/FHIR/operations.html Binary data with FWoA: https://github.com/awslabs/fhir-works-on-aws-deployment/blob/83127bee1c0850a7bdb670a36afe8c5e4cfb023d/README.md#accessing-binary-resources

Meanwhile, the FWoA UI was built for demoing purposes and do not support all the functionalities available to FWoA. To fully explore the capability of FWoA, I would recommend setting up the Postman collection to trigger FWoA API directly.

Thanks, Yanyu

youbenuse commented 1 year ago

Hi, @Bingjiling

Thanks for your kind explanation. First of all, I'm using Postman Collection and modified WoA UI to create many additional resources such as Organization, Encounter, Location, Patient, Practitioner, Observation, etc...

And I just checked Binary endpoint that I can upload file, by the way, This api related with Clinical Document? The unclear thing is that, I can't understand how can get/post Clinical data using FWoA.

Also, FHIR has built-in option such as Accept in the API request header to allow user can post xml type data as application/fhir+xml and _format query type such as _format=application/fhir+xml or json, etc...

There is no this kind option in FWoA? I already tried to do this on the postman but It's not working.

And regarding IGs implementation guide, I don't understand how complied implementation IGs applied to the current FWoA server. That's why I'm not sure my compiled IGs is working properly or not on the FWoA Server side. If you have a good experience for any IGs Implementation, would you like to explain how it works?

Hope to get your kind explanation. Thank you so much.

Youssef.

nguyen102 commented 1 year ago

Hi @youbenuse,

I'll answer your questions below.

  1. FWoA only supports JSON data types.
  2. FWoA support searching for new fields defined in IGs. When an IG is compiled for FWoA, a search mapping is generated. FWoA use that mapping to allow clients to search for the fields defined by that IG. FWoA does not make any changes to FHIR resources that are created. Resources are stored as is.

Does the CDEX and CCDA IG expect resources to be transformed in a certain ways when certain resources are stored in FHIR?

-Tim

youbenuse commented 1 year ago

Hi, @nguyen102

Thanks for your kind reply.

As you can see following CcdaOnFhirServer.openapi.json from CCDA IGs pakcage, this enable to make api request as application/fhir+xml.

image

Even, all of test public FHIR server allow to call api endpoint as Accept Header application/fhir+xml type. Such as HAPI and Firely Server https://confluence.hl7.org/display/FHIR/Public+Test+Servers

So I'm not sure why FWoA not support xml type and want to know which kind of server we are using on the FWoA? If yes, Above test servers are already implemented CDEX or CCDA IGs?

If you check hl7.fhir.us.core IG, it's require both xml and json type. Also, hl7.fhir.us.ccda is require both hl7.fhir.r4.core and hl7.fhir.us.core as dependecies.

So if we compile IGs on the FWoA it's only validate API request url endpoints and their params as routing and not update request data type? As you know, CCDA and CDEX is transform data type on the server side.

The most important thing is that CCDA is IG which one exchange XML data type to FHIR format. So I want to know how can I implement CCDA IG to enable xml format on the FWoA server.

Looking forward to hearing positive support.

Best Regards. Youssef

nguyen102 commented 1 year ago

Hi @youbenuse,

We are working to add more features to FWoA over time. As of now we don't support fhir+xml or transforming data types.

You can submit a feature request using this template.

We also welcome contribution to the code base. If you have time and would like to help, please consider contributing.

-Tim

youbenuse commented 1 year ago

Hi, @nguyen102

I already created feature request.

But regarding contributing, I need to know about your current aws-routing management and IG implementation logic. As well as, I need to know what library do I need to check to enable XML data type from API request.

So hope to inform this kind of information to check on my side for the best solution...

Looking for your more details answer about my previous and above questions.

Thanks Youssef

nguyen102 commented 1 year ago

Adding feature request to link these two issues together. https://github.com/awslabs/fhir-works-on-aws-deployment/issues/808

nguyen102 commented 1 year ago

@youbenuse

When you say "I can't understand how can get/post Clinical data using FWoA.", are you referring to Clinical resource types? There's a list of resource types here and Observation is one of the resource types listed as Clinical. If you make a POST request to <fwoaApiServer>/Observation, you'll be able to store an Observation record in FWoA.

You can then get the observation record by making a GET request to <fwoaApiServer>/Observation/:observationId.

Before updating the code to support XML, I would check that the IG is working as you expect. Are the requests you're making in XML format? If they are, you can convert xml to json using this tool. Once your convert your XML request to JSON, you can make requests to the FWoA server. You should check that FWoA returns the expected response given the request that you make.

If the response is as you expect, you can consider implementing XML support for FWoA and we'll provide guidance on how that can be done.

youbenuse commented 1 year ago

Hi, @nguyen102

Yes, I already implemented FWoA Server and UI successfully and can create all of resources such as Observation, Patient, Practitioner, Encounter, Location, etc... Also I can test everything on the Postman and UI side that I customized for create resources and trigger bulkexport data.

But as I mentioned, CCDA is support both application/fhir+json and application/fhir+xml request type.

So I want this option to create resources using xml data or can get xml data type api response...

But I'm not sure how can I implement IGs and if we implemented IGs correctly, we still need to update lambda api request acceptance data type?

This is my main point which I want to know in more details.

Thanks Youssef

youbenuse commented 1 year ago

@nguyen102

Would you like to correct me with following things asap?

Thanks Youssef

nguyen102 commented 1 year ago

Hi @youbenuse,

How can I implement CCDA IG correctly? You can follow the instructions here to deploy FWoA with the IG you're interested in. Once you deploy FWoA, I would recommend you confirm the IG work correctly by checking that FWoA returns the correct responses given CCDA specific search requests.

How can I enable XML data format on the FWoA? FWoA does not support XML data. This will need to be implemented. I'll provide some details on how XML can be implemented. At a high level you need to convert the XML request to JSON and then you'll need to convert the JSON response to XML. This can be done in the routing package where the routing is set up here. You can consider using an express middleware to change the request. Here's an example on stack overflow. You can also use an express middleware to change the response. Here's an example on stack overflow.

youbenuse commented 1 year ago

Hi, @nguyen102

Thanks for your more detailed information. But is this best solution to enable XML data type on the FWoA?

As you know, all of public FHIR test server can get XML data type from API request and respond with XML data type. If we convert XML data type as express middleware, It means FHIR server only accept json data type.

In this point, I'm not sure what kind of FHIR server we are using in FWoA.

If this is a best choice to enable XML data type, how can I update routing package code base can I create commit my change in this repo directly?

Looking for your kind reply asap. Thanks Youssef

nguyen102 commented 1 year ago

Hi @youbenuse,

See the answer to your questions below:

But is this best solution to enable XML data type on the FWoA? yes

If we convert XML data type as express middleware, It means FHIR server only accept json data type. FHIR server will store data as JSON but will be able to accept XML data type and return XML data type

In this point, I'm not sure what kind of FHIR server we are using in FWoA. FWoA is a FHIR server. You can see the type of FHIR server it is by checking it's capability statement at <API-URL>/metadata

If this is a best choice to enable XML data type, how can I update routing package code base can I create commit my change in this repo directly? You can fork this repo and the routing repo, to make the necessary changes.

Development instructions are here

github-actions[bot] commented 1 year ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

youbenuse commented 1 year ago

Actually, This is not best option to add custom middle ware on the routing repository.

Because post .xml data type is arrived as Buffer on the express server if we don't use xmlparser. Also, if we use xmlparser, request header and other parameters will be empty on the server side.

And another critical thing is that there is no professional xml to json parser npm package that can parse into correct json file from .xml file type.

And then, we need to validate request header which is including 'Accept' header option as xml type. Also, we need to update response middleware to update json to xml type for this...

As last one, I'm not sure how can check IGs are implemented on the FWoA.

In my understanding, many public test FHIR server able to submit XML or JSON datatype to create resources... like this... https://hapi.fhir.org/baseR4/Patient

image

nguyen102 commented 1 year ago

Because post .xml data type is arrived as Buffer on the express server if we don't use xmlparser. Have you tried using https://www.npmjs.com/package/express-xml-bodyparser? There's an example SO answer here

As last one, I'm not sure how can check IGs are implemented on the FWoA. and In my understanding, many public test FHIR server able to submit XML or JSON datatype to create resources... like this.. are answered here and here

youbenuse commented 1 year ago

As I mentioned, if we use xmlparser all of request headers are empty then it will not work anymore on the FWoA.

And due to my understanding, CCDA IG is support XML parse to JSON. Then how can we check it in this FWoA?

nguyen102 commented 1 year ago

As I mentioned, if we use xmlparser all of request headers are empty then it will not work anymore on the FWoA. FWoA uses Express for routing, so it might be helpful to Google for XML support for Express server. I found this blog post here. If you're unable to find a solution for XML header support with an Express server, you can try building a proxy in front of FWoA. The proxy will expose APIs that accepts XML requests. The proxy converts the XML request into JSON and pass the request to FWoA.

And due to my understanding, CCDA IG is support XML parse to JSON. Then how can we check it in this FWoA?

I'm not familiar with CCDA IG, but couldn't you use a tool like this to convert XML request/response to JSON? You can convert sample XML requests to JSON and use the new JSON request with FWoA.

youbenuse commented 1 year ago

Hi, @nguyen102

Thanks for your kind support. But I think you are missing this point...

Here is CCDA XML data example.

image

As you can see, This resource type is new in FWoA. And as I told you, I compiled CCDA IGs and deployed to FWoA according to the Implementation Guide. But there is no way, How can I test FWoA is accept CDA XML datatype on the server side. As I know, FWoA not support XML data type so I implemented custom XML format to JSON but it's only for existing resource types. In this case, how about above CCDA ClinicalDocument Type on the FWoA.

Hope to get FWoA's more effective solution to implement this. Thanks

Bingjiling commented 1 year ago

Hi @youbenuse,

Thanks for the response!

Could you please explain how you implemented custom XML format to JSON functionality? Also can you show full request and response data for posting resources(both existing type and CCDA ClinicalDocument Type) using XML after the implementation?

Thanks, Yanyu

youbenuse commented 1 year ago

Hi, @Bingjiling

I can't create PR for my XML format to JSON custom middleware on the routing repo. Because I was urgent to add this option so I hard coded on the routing node_module directly.

Speaking about my process, I added custom middleware to check request header includes accept:application/fhir+xml and get if exist, parsed xml to json using xml2js npm package.

But it was a challenge to convert exact json resource type for FHIR. So I'm trying to figure out better way...

And regarding CCDA, even I'm not sure how can I test CCDA IG implementation result on the FWoA. As you know, we have postman collection for each resource types. Once I deploy compiled CCDA IGs to FWoA, it's generating new API endpoint routing? I want to know about this in more clearly that I can understand how IGs applying to the current existing FWoA server.

Thanks

nguyen102 commented 1 year ago

Hi,

Our team does not have any experience working with the CCDA IG, so we do not know what the requirements of CCDA IG is. We can tell you how FWoA works and the steps for testing a generic IG, but it would be up to you to confirm if the specific IG you're interested in works correctly.

  1. Can you link us to the IG file that you're using to compile the IGs? All definition files in the IG needs to be JSON files and there needs to be an .index.json file. If they are not JSON files the IG will not compile correctly.

  2. You can check the FWoA capability statement to see if FWoA support the new resource you're interested in. In this case, it looks like you're interested in ClinicalDocument. The url for that is {{API_URL}}/metadata.

  3. You need JSON examples of CCDA resource types that you can use for testing. Those samples need to be in JSON. If they're in XML, convert them to JSON so you can test it against FWoA. The route for creating a new resource is POST {{API_URL}}/<ResourceType. In your case it might be POST {{API_URL}}/ClinicalDocument. If creating the resource works, you'll want to test reading the resource by using the GET API.

  4. You mentioned you implemented XML to JSON support for existing resource types. If done correctly, it should technically work for all resource types. Can you describe what you did, and why it works for existing resource type, but not new resource types?

To repeat, we have not worked with CCDA IG before. We would need to rely on you as the CCDA IG subject matter expert to let us know what the requirements of that IG is.

To answer the specific question you asked: Once I deploy compiled CCDA IGs to FWoA, it's generating new API endpoint routing?

-Tim

youbenuse commented 1 year ago

Hi, @nguyen102

Thanks for your kind explanation. Regarding my CCDA IGs implementation, I think I compiled correctly this IG after I download package which is including .index.json file. Also, I think I already shared my CCDA IGs in above topic session. image

To confirm IGs implementation correction again, what kind of change do I need to made in IGCompiler.test.ts? And this is CCDA document links. http://hl7.org/fhir/us/ccda/index.html https://confluence.hl7.org/display/SD/CCDA-on-FHIR

And regarding xml format customization, I got your suggestion regarding external library to convert XML to JSON. https://fhir-formats.github.io/# Is there anyway, that we can use this library on the FWoA? My customization is not working properly for all of resource types, because all of open source libraries that I tried are not working properly.

Also, your comment was helpful to test my IG implementation result, so I'm gonna try to test it by convert to JSON after check metadata response.

Hope to have more detailed discussion furthermore. Thanks

nisankep commented 1 year ago

FHIR Works on AWS has been moved to maintenance mode. While in maintenance, we will not add any new features to this solution. All security issues should be reported directly to AWS Security at [aws-security@amazon.com] (mailto:security@amazon.com). If you are new to this solution, we advise you to explore using [HealthLake] (https://aws.amazon.com/healthlake), which is our managed service for building FHIR based transactional and analytics applications. You can get started by contacting your AWS Account team. If you are an existing customer of FHIR Works on AWS, and have additional questions or need immediate help, please reach out to fwoa-migration-support@amazon.com or contact your AWS Account team.