Expand jsonkeyvalues to support variable replacement across initializer domains

[mseaton]

As initializer use has expanded, and particularly as it has become the core component of content packages in OpenMRS, a challenge exists in how to share or re-use content packages across implementations that may not have exactly the same expected configuration.

For example, let's say that I have a large set of configurations across all domains that I have packaged up to support a particular country or program area, and I want to roll this out to dozens or hundreds of facilities consistently. But on a handful of these systems, the uuids for a particular encounter type or location are different than expected.

Let's also say that some uuids are used across many different domains and need to be kept in sync with each other. Maybe you have encounter type or location uuids that need to be set consistently across metadatatermmappings, globalproperties, htmlforms, etc.

To help address these challenges, my proposal is that we leverage the existing jsonkeyvalues domain, and to allow any key/value pair defined to represent a variable that could be replaced in a text-based configuration file prior to loading.

1. Move jsonkeyvalues domain to be the first domain processed
2. Support a special "_order" key in the domain to allow control over which files are loaded in what order
3. Load all of the jsonkeyvalues into memory, allowing higher-ordered files to override values with the same key
4. Add support when loading any text configuration file (csv, xml, json, txt, yml, yaml, etc) to replace any variables based on a standard syntax, eg. ${key} or {{key}}, etc.

This could potentially be made optional/configurable to guard against possible scenarios where this replacement is not desired via a configuration property.

@mks-d / @rbuisson / @Ruhanga / @ibacher / @mogoodrich - thoughts welcome.

[mseaton]

An alternative variation on the above, if we don't want to use the existing jsonkeyvalue domain, would be to add a new domain called "constants" or some better name, which could be used for a similar purpose. A benefit here is that a new domain would not necessarily have the same constraints on it as the jsonkeyvalue domain, and would also be 100% backwards-compatible as there is no existing utilization we need to consider.

[ibacher]

First, I love the idea of this feature, but I think it's execution might be a little difficult, depending on how much of this ambitious vision we want to implement.

Basically, I think this can be straight forwardly added to CSV domains, but it becomes much trickier with non-CSV domains, because there is much less consistency with how they are processed.

JSON:

• jsonkeyvalue: obviously we don't want this
• forms: all the JSON form domains are read using Jackson and we could write a custom deserializer for Jackson to handle this (this would also allow us to build out support for non-JSON domains since Jackson supports way more formats than we need
• disposition: these files are not processed at all, and adding processing is probably non-trivial since we either need to change the domain to overwrite the source file or change how it functions altogether

XML:

• liquibase: I don't actually think we want to apply this here (we only look for "liquibase.xml" and we do not currently do anything with it other than pass it to Liquibase)
• GPs: we currently use XStream; we should swap to Jackson
• addresshierarchy: This isn't processed by Iniz at all; it just calls the module
• HFE: this is parsed by HFE's own tools, but there is some post-processing we could use to substitute things

Properties:

• messageproperties: this is not really a standard domain in that everything is read by Spring in @PostConstruct initializer. This basically means that we cannot depend on any Iniz domain being loaded before it. I don't know if this is one of the domains we were planning on including...

I don't think there's anything wrong with leveraging jsonkeyvalues. The domain basically just loads files and sticks their contents in a big map.

[mseaton]

I wasn't really thinking of having to change any deserializers, but rather just change the raw content pre-processing. The bigger issue to overcome is in identifying whether files have changed based on their checksums, as a change to a property in jsonkeyvalues that is referenced in another config file would need to trigger a change.

Basically, I'm proposing that we would first load file contents in. If these file contents are considered text (hopefully a standard library can help us identify this), then we get the contents as a String, and iterate over the jsonkeyvalues that have been loaded, applying them to the source String to produce an altered target String.

We then take a checksum of this altered string, and proceed as normal initializer loading from these pre-processed file contents.

[Ruhanga]

To help address these challenges, my proposal is that we leverage the existing jsonkeyvalues domain, and to allow any key/value pair defined to represent a variable that could be replaced in a text-based configuration file prior to loading ...

The jsonkeyvalues domain seems like the best fit indeed.