#256 - Process substitution overrides in order

JettJones commented 3 years ago

Fixing #256 - and a small radius around it.

The Scenario

Using hocon in a larger system where this came up.

keys: { something: "${lookup}-${name}" }

overridden by:
keys: { something: "${lookup}-legacy-name" }

During substitution & replacement, the following happened:

Starting queue had these substitutions: [${lookup}(parent), ${name}(parent), ${lookup}(child)]

Process ${lookup}(parent), the result value still had -${name}, so this substitution stays in the set to re-process, and enqueues ${name}(2) at the end of the list.
Process ${name}(parent) - this produces a static result string, and is removed.
Process ${lookup}(child) - this produces a static result string, and is removed. (that's all the substitutions, so the loop restarts)
Process ${lookup}(parent) - overwrites step 3. No overrides remain, so is removed.
Process ${name}(2) - also overwrites step 3. No overrides remain, so is removed.

Proposed Fix:

When looking at a substitution, first check if it has a parent override, and if that parent is active in the substitution list. Wait for all parents to complete before processing the substitution.

This caused two test failures:

g: ${?missing1} ${?missing2} would be alternatively added or removed from config as the substitutions processed. This diff changes _do_substitute to only write/add a value if the value is not None.
At step 3 above, if lookup is not processed, this is detected as a config loop, because the set of substitutions to evaluate is the same after the loop (${name} is added once and removed once). To get around this, I added a second check to avoid re-queueing substitutions that are already being processed.

Testing

[x] Added new test case
[x] Passed all tests (except NamedTemporaryFile)
[ ] NamedTemporaryFile

^ NamedTemporaryFile does not work on Windows (python.org) - I'll try to kick the tires in docker if there's traction for landing this.