Schema upgrade process improvements

[feerrenrut]

1. Upgrade step unit tests

As a developer writing config upgrade code, I want to be able to define the cases to be tested automatically. So that my development and testing experience is faster and less error prone. The initial work for the upgrade process was done in #6558

Some things to consider:

• Testing several cases for a new upgrade step is time consuming, and error prone.
• Any future work in this area can not easily see the cases that must be considered and why.
• We do not require config upgrade code for added keys, only deleted keys, or keys that have changed meaning. Typically keys that have changed meaning would have their name changed also. Effectively this is a deletion, introduction of a new key, and a migrated value. The migration of the value may also involve some transformation.
• When writing to a config key that is not defined in the schema, there is no error. We rely on this in some cases. An automated test that ensures that the schema contains the same key as is referenced from the upgrade step could help to catch errors here. One potential error would be a misspelling of the key in the schema. This could be hard to catch by code review.

Proposal: Introduce a unit testing framework / helper that could allow a developer to specify a dictionary of the keys and values of interest to be upgraded, and the expected result. This would allow the developer to specify testing permutations easily. This test should also check that keys referenced do indeed exist in the schema.

2. A task / migration based upgrade process

As an NVDA maintainer about to incubate config upgrade code, I don't want to have to check if a particular schema version is already used and incubated (but not included in the candidate code). This is to make the code review, incubation, merge to master, and any backouts faster and less error prone.

• The order that config upgrade steps occur should be "well defined". That is, the order should be explicit, and consistent.
• It should be easy for a developer to check what version the config is at, and what upgrade steps have been applied.
• We want to make it hard / impossible to have a version value collision, as is currently the case. One developer, bases their config upgrade code on a version of master, while there is config upgrade code currently being incubated. In this case the version number value is dependent on knowing this.
• If config upgrade code needs to be backed out of next / master, then version numbers need to be updated. Currently version 2 must follow 1, there is no skipping straight to 3.

Proposal:

• Rather than use a linear progression of integers for the schema version, use an ordered list of migrations need to be applied to the config. These migrations can have a human readable name, probably the name of the migration step function (since that saves us having to have a mapping).
• The users config file will contain a list of migrations that have already been applied, the order of this is not important.
• To ensure there are no naming collisions (for the migration step names), the upgrader can ensure that there are no duplicates in the migrations list.

Some questions:

• Its rare that there would be dependencies in these steps, but it is possible. How would we handle this? Simple solution may just be an inline comment on the "ordered list of migrations" that specified that this step depends on step x.

[feerrenrut]

@jcsteh Any thing to add?

[Adriani90]

@feerrenrut is this partly solved with the new automated testing framework?

[feerrenrut]

@Adriani90 No. Though, if there were dependencies on testing frameworks when this issue was created, they are definitely satisfied now. The problems described in this issue have not occurred yet (that I know of), and thus it might not be pragmatic to spend time on this.