This ticket is an epic to design and implement a standardised and scalable storage solution for our primary data.
The changes will be fundamental to the product but are critical for any production use of the product.
Hearth is no longer officially maintained by Jembi Health Systems because they have not been able to secure enough funding to do so.
Apart from Hearth, there is no OpenSource NoSQL, and in our opinion, scalable FHIR server that suits our needs. We do not recommend the use of HAPI-FHIR at scale because PostgreSQL cannot be horizontally scaled. It can only be vertically scaled. This has a cost implication to implementing governments.
However, only 2 of these are actively maintained projects and only 1 of them is maintained by a recognisable entity.
We are not suggesting we use any of these for the following reasons:
1) we do not want to be dependent upon a small, private organisation or private individual to maintain our database server that is not a core partner. We would need to perform a thorough analysis of their longevity and formalise a relationship that we would have with such an organisation. None of these orgs are comparable to a MongoDB, Elastic, Minio or InfluxData where we can depend on their size in the market.
2) only one of them shows us a validated schema that can be sharded (horizontally scaled) but that only supports 1 FHIR document type - The Patient type.
3) all seem to follow a Hearth-like SQL schema approach to NoSQL data storing. We want to combine some FHIR document types into a single schema for faster performance to utilise the true power of NoSQL.
However, there are aspects of these projects which along with Hearth can inspire us as we plan our own approach for best cost & performance in the civil registration context.
In OpenCRVS v1.0 we made our own fork of Jembi's Hearth and maintain security upgrades.
At some point in the future, we plan to gradually phase out FHIR completely from our storage, starting with registration collections, e.g.: Observation Encounter etc and instead expose a FHIR API.
Enable offering data in other data models such as OMOP
Increased team velocity
Less code to maintain – reduced total cost of ownership
Long-term strategy:
Create a new service: records. This service will be the main storage service for vital events and all related data.
Design our own simplified civil registration specific data model
Gradually migrate stored objects and server side logic to use the new service
Immediate noticeable performance boost to features we migrate
What we plan to perform when gradually phasing out Hearth:
We want to maintain all the existing routing request URL query param fhir API functionality
We should not have a separate Composition, Patient, RelatedPerson, Task, Observation, DocumentResource and Encounter collection, but instead a single Registration collection that contains a simplified version of a fhir.Bundle of all of the above.
We should maintain separate Location, Practitioner, PractitionerRole collections.
The Registration collection should make use of Mongo indices so that it can be sharded and horizontally scaled. Maybe the Registration collection needs to exist in an entirely different database or volume?
We want all the code to be TypeScript
Hearth has a customisable "plugin" architecture. So that it can be extended by other organisations using it as an OpenSource lib. We have no intention of extending beyond the plugins we need, so we can deprecate this as long as we include the checkDuplicateTask functionality somehow.
It should be protected by the same JWT auth as the rest of our app and thus be penetration tested. Only Location GET endpoints should be auth:false
We should maintain the existing FHIR validation functionality from Hearth
This ticket is an epic to design and implement a standardised and scalable storage solution for our primary data. The changes will be fundamental to the product but are critical for any production use of the product.
More details coming later..
We have forked Hearth and maintain it here.
Hearth is no longer officially maintained by Jembi Health Systems because they have not been able to secure enough funding to do so.
Apart from Hearth, there is no OpenSource NoSQL, and in our opinion, scalable FHIR server that suits our needs. We do not recommend the use of HAPI-FHIR at scale because PostgreSQL cannot be horizontally scaled. It can only be vertically scaled. This has a cost implication to implementing governments.
Interesting reading and applies to hapi-fhir: https://vneilley.medium.com/most-fhir-servers-are-unusable-in-production-8833cb1480b1
We have discovered these projects ...
https://github.com/icanbwell/fhir-server https://github.com/bluehalo/node-fhir-server-mongo https://github.com/bluehalo/node-fhir-server-core https://github.com/Chinlinlee/Burni
However, only 2 of these are actively maintained projects and only 1 of them is maintained by a recognisable entity.
We are not suggesting we use any of these for the following reasons:
1) we do not want to be dependent upon a small, private organisation or private individual to maintain our database server that is not a core partner. We would need to perform a thorough analysis of their longevity and formalise a relationship that we would have with such an organisation. None of these orgs are comparable to a MongoDB, Elastic, Minio or InfluxData where we can depend on their size in the market.
2) only one of them shows us a validated schema that can be sharded (horizontally scaled) but that only supports 1 FHIR document type - The Patient type.
3) all seem to follow a Hearth-like SQL schema approach to NoSQL data storing. We want to combine some FHIR document types into a single schema for faster performance to utilise the true power of NoSQL.
However, there are aspects of these projects which along with Hearth can inspire us as we plan our own approach for best cost & performance in the civil registration context.
In OpenCRVS v1.0 we made our own fork of Jembi's Hearth and maintain security upgrades.
At some point in the future, we plan to gradually phase out FHIR completely from our storage, starting with registration collections, e.g.: Observation Encounter etc and instead expose a FHIR API.
Benefits are:
Long-term strategy:
What we plan to perform when gradually phasing out Hearth: