Bulk import of FHIR Resources

[Rkareko]

There is a use-case for importing bulk data from an external data source. There are around 8k Patient resources each having 5 or more related resources. The data is provided in form of a file containing a JSON Array of FHIR resources.

The current publishing tool has a few limitations : -

1. The tool ingests data in form of files containing a single resource. To handle this uses case a pre-processing stage is required to extract the data into different file.
2. The tool consolidates all the FHIR Resources into a single Transaction Bundle. This is a bottleneck when dealing with large amounts of resources.

[Wambere]

The importer tool is better suited to do bulk imports, the Publish command on the other hand is more suited for publishing app configurations

I think the fastest way to handle this would be to add functionality into the importer that:

• reads the file provided containing a JSON array
• processes the data in chunks
• build a payload and post to the server

LOE about 4 days

[Wambere]

@Rkareko @allan-on just a heads up that the LOE for this might increase based on this comment

The current implementation can be used to test sorted datasets or sets with no references, or maybe we can play around with the chunk sizes to ensure it works? as we figure out how sort the whole file