[DISCUSS] Kibana index version migrations

[stacey-gammon]

Proposal

Introduce a testable migration process that allows developers to incrementally add complex migration steps throughout the development of several minor releases.

Goals

Testable The process should be easily testable so that at any point a failure to account for a required migration step will be captured. Ex: I almost submitted a PR that removed a field from the kibana mapping. Without a manual migration step, that data would have been lost. This would have gone unnoticed until late in the 7.0 index upgrade testing (if at all).

Incrementally add migration steps We don't want to push the entire migration burden to the last minute. We should be able to incrementally add migration steps and tests to catch issues early and prevent major releases from being pushed back due to last minute bugs being found.

Flexibility Right now we assume the entire re-indexing will fit into a painless script, but this won't hold us over for long. As a specific example, I'd like to migrate some data stored as JSON in a text field. Manipulating JSON from painless is not possible currently. I'd bet even more complicated scenarios are right around the corner. Our approach should be flexible to easily accommodate migration steps that won't fit into painless.

Make it easy for developers Our approach should be un-intimidating so all developers on the team can easily add their own migration steps without requiring too much specific knowledge of the migration code. Making this a simple process will encourage us to fix issues that rely on .kibana index changes which can help clean up our code. There have been outstanding issues for many months that don't get addressed because they require mapping changes. A very small change (the same PR mentioned above) that incidentally removes the need for a field on the kibana mapping, and a pretty straightforward (albiet still not possible in painless) conversion, should be easy to throw into the migration.

Single source of conversion truth There are potentially different areas of the code that need to know how to handle a bwc breaking change. For example, I'd like to introduce a change in a minor release which removes the need for a field on the .kibana index. In order to support backward compatibility, I need to do three things:

• Make sure objects exported prior to the change can be imported and look as expected.
• Make sure objects stored in an index that hasn't been migrated can be opened and look as expected
• Make sure objects in a migrated index can be opened and look as expected.

I'd have to think more about how/whether all three spots could take advantage of the same code.

Pluggable? We might want to make this pluggable, so saved object types we are unaware of can register their own migration steps.

Questions

• Should this process focus only on migrating saved objects, the kibana index as a whole, or cover both?

Implementation details

TODO: have not thought this far ahead yet.

cc @epixa @archanid @tsullivan @tylersmalley

[epixa]

If you have 2+ Kibana servers, one will run migrations when it starts. The other(s) will not run migrations, as migrations will already be running (in one of the other servers). So, what should these secondary servers do?

A holding pattern probably makes sense. In a lot of ways, it's the same thing as the one running migrations, it's just not actually running them itself. It could still serve a "migrations running" response to http requests or something, just like the server that's doing the migration.

I think we need to move mappings over from original indices, but trump them w/ mappings defined by current plugins. ... Is there a better alternative?

I can't think of one off-hand. This seems like an appropriate solution - basically we're just moving the problem down the road. I do think we should be validating the core shape of these orphaned mappings though. There are assumptions Kibana itself makes about the shape of all documents in its index, with things like a keyword type attribute and an id. While we can't validate the plugin-specific changes, we can certainly validate our core assumptions. What do you think?

[chrisdavies]

@epixa

I think the holding pattern makes sense.

I agree that we need to move the mappings.

I've got mixed opinions about validating mappings / docs. On the one hand, I like having as strong a guarantee as we can that the index is in a state we understand. On the other hand, what do we do if we find mappings / docs that don't meet our expectations? Fail the migration? In that case, how does a customer fix the issue? Should we add a --drop-invalid-docs option or something along those lines?

[chrisdavies]

Jotting down notes from a conversation w/ @tylersmalley and @spalger.

We've decided to explicitly not support deletion of the .kibana index. Migrations ensure the index is created and properly aliased when Kibana starts. This new behavior disrupts any existing code that would have deleted the index (as you can't delete an alias in the same way you would an index).

Instead, we are putting logic into the saved object client which will assert a valid index migration state prior to each write, and fail the write if the index is in an invalid state. This seemed like a reasonable approach to us, as by definition, if people are tinkering with the Kibana index and get it into an unknown state, it's unknown what the consequences of automatic recovery would be.

[chrisdavies]

This feature is currently in code-review, finalization mode, but I've come across a potential roadblock.

Problem

The current implementation will require

• Update of all test fixtures
  • Every time a migration is added (even if it doesn't affect the docs in the fixture)
  • (Maybe) every time the Kibana version bumps (the index names contain the Kibana version)
• The sheer number of breaking tests is a yellow (or maybe red) flag
  • If it takes this much work to get tests working, how will this change affect 3rd parties?

Possible solution

I think it might be possible to have a migration system with a lighter touch and still hit the goals we've set. We could create a system that behaves similarly to the current Kibana version:

• No migration state per index (except a temporary doc used for optimistic concurrency)
• No alias
• No index creation at start up-- only migration of existing indices
• Saved object client puts an index template prior to any write operations
• The Kibana index can be blown away and re-created by tests

How it would work:

• We'd add a "migrationVersion" property to each doc (see details below)
• When Kibana starts, it makes sure all existing docs are up to date
  • It does this by querying for any docs which have an out of date migrationVersion
• The saved object client will also transform any docs it reads or writes
  • This allows us to handle scenarios where docs are snuck into the index after migration (e.g. the way our tests do)
• Import / export is simplified, as the version info is exported / imported with each doc and properly handled by the saved object client
• If a document has migration info from an unknown plugin (either missing or disabled)
  • We will still persist it, and allow mappings to dictate whether or not the persist succeeds
  • When (if ever) that plugin is enabled, the docs will be migrated if necessary
• Migrations could be done in a (somewhat) non-destructive way without requiring an alias (if we so choose)
  • Reindex to a backup index prior to migrating the current index
  • Migration of the current index is a simple read, transform, update of each out-of-date doc
• Initially, we can avoid supporting having "seed" migrations, but these are fairly simple to add back in if / when needed
• The migrationVersion would look something like this:

{
  migrationVersion: {
    pluginA: 3,
    pluginB: 1,
  },
}

• Explanation:
  • The pluginA properties are the ids of any plugins which have data in the document (generally, there will be only one)
  • The numeric value is the index of the latest migration from that plugin which has been applied to the doc

[epixa]

@chrisdavies What's the best way for me to learn about the current state of migrations, so I can better parse your new proposal? I'm looking for a definitive technical overview of how migrations work. Would that be https://github.com/chrisdavies/kibana/blob/feature/migrations-core/packages/kbn-migrations/README.md?

[chrisdavies]

@epixa That readme is currently the best written source outside of the source code itself, so yep! I'm also happy to hop on Slack / Zoom as necessary.

[chrisdavies]

Chatted w/ @spalger about this. Here's the summary of that conversation:

• Not tracking index-level migration state, but per doc
• When we start up, we do a query to migrate if needed
• This is also done at read time and write time
• Saved object client's find operation is a special case.
  • We'll add an aggregation to check if any docs are in old state
  • If so, we'll do a de-duped migration (de-duped per Kibana instance)
  • Re-executes the find and returns the result after migration completes
  • If migration fails
  • Kibana should go into red-state
  • Log the id of the doc, the id of the plugin and the id of the migration and the error
• Saved object will continue to function as it does today w/ exception of the migration version check

We both think this change is worth doing now rather than moving forward with the current approach. This per-document approach is more robust in a number of ways, and also eliminates a good deal of complexity (such as the need for a --force option, the need to update test snapshots all over the place, etc).

[chrisdavies]

Chatted w/ @tylersmalley and @jbudz and debated the pros/cons of the various approaches. The decision was to stay the course and fix all of the broken tests, as the current approach has less impact on performance, and ultimately is architecturally simpler even though it's significantly more code.

[epixa]

In a world of automatic migrations, the whole premise behind esArchiver seems flawed. Or at least the extent to which we're relying on esArchiver for tests is flawed.

Rather than importing raw document data through elasticsearch, we should probably be importing saved object data through kibana instead. If I add a migration and a test fails, then that should indicate a regression, but instead it's going to be treated as a blanket reason to rebuild the fixture data with esArchiver.

I can see esArchiver fixtures being useful for testing our migration system itself and/or for testing Kibana running against old known legacy document states if that isn't handled by the migration system automatically. But beyond that all other tests should really be seeding data the way we expect data to be seeded in the real world.

[chrisdavies]

@epixa You and Tyler are on the same wavelength and have convinced me. :)

[chrisdavies]

Good news, @tylersmalley ! More changes. @epixa and I talked this morning, and came up with a scenario that is going to come up in the next release. That scenario is splitting plugins apart into smaller plugins, and changing what plugins own a type.

This pointed out a flaw in the current implementation, and a change that both facilitates this requirement while also simplifying the implementation.

Currently, we allow migrations to apply to any doc based solely on the filter the migration specifies. We track migrations per plugin, and enforce migration order on a per-plugin basis.

Changes

• Track migrations on a per type basis
• Changing the type (e.g. dash -> dashboard) can be done by creating a migration for "dash" that returns a doc of type "dashboard"
• Type ownership can be moved between plugins so long as the migrations also move
• I think this means we no longer need to store mapping information in migrationState itself

[chrisdavies]

@epixa pointed out that the current direction of migrations would allow us to easily make breaking changes to our API, as migrations would make it easy to change the shape of our data willy-nilly, and right now, our raw data is our API in some sense, as we expose it via the saved client system.

So, we decided to disallow breaking mapping changes during minor version transitions, and relegate such changes to the major version boundaries. Minor versions would allow adding properties to mappings, but not removing properties, nor changing leaf-properties.

@epixa suggested that in minor versions, we don't migrate the index at all, but simply run a transform function (one per type) on all docs coming into / out of the saved object API. So, the index may end up with docs of varying versions, but at read-time, they'd get sanitized and made consistent.

After heading down this road a bit, I think I've come up against a roadblock:

Problem 1

• We can only run transform functions on full documents (e.g. not on partial documents)
• There appears to be at least one saved object method that produces partial docs (find, and maybe update)
• This means that any attempts at making the data consistent at read-time are not going to be fully reliable

Problem 2

• Are writes backward compatible, too?
  • If so, how do we know which field wins?
  • Say, we have added a 'foo' field which is hydrated from a JSON blob
  • If the foo field doesn't exist, we hydrate it from JSON, if it does, we hydrate the JSON from the foo field
  • Some 3rd party reads that doc (including the foo field they know nothing about), they update the JSON as they always have, and pass the object back (including existing fields, such as foo, which they know nothing about), but their update gets obliterated by the foo field!

Possible solutions

Migrate the index even on minors

• This allows us to do finds predictably
• Instead, we can go back to migrating the index at startup, which for minor versions would look like:
  • In the _meta field, store
  • Kibana version
  • Checksum of all the types that have transforms defined (allows us to handle plugin enable / disable)
  • When booting, after Elasticsearch becomes available:
  • Compare the _meta with the current Kibana values
  • If index.version has a different major version than kibana.version, throw error
  • If index.version is more recent than kibana.version, throw error
    • Maybe allow starting w/ a --downgrade option or something, which allows downgrading within the same major version
  • If the version diff is a minor version bump,
    • Patch the mappings and index template
    • Scan docs in, run them through their transforms, write them back to the same index
    • When done, mark the index as up to date w/ kibana.version
• In export files, we need only store the Kibana version, and can either store it once, or per doc, whichever is easiest
  • Importing will call the saved object API, passing it the doc(s) along with the Kibana version stored in the file
  • The appropriate transform(s) will be applied based on the Kibana version
• When the saved object write APIs are not passed a Kibana version, they will apply only the current major version transform per doc
• When developing a transform,
  • The developer will need to modify the Kibana version to be the appropriate major version for their transform (e.g. 6.x, if developing a 6.x transform)
  • In master, package.json is currently 7.x, so anyone writing 6.x transforms will need to change that temporarily

Modify saved object find to not support source filtering

• Here, we'd always fetch full documents in the find API
  • Run them through the transforms
  • Perform source filtering logic on our own in Node
  • Pass the transformed and source-filtered documents over the wire
• My concern with this is that the added complexity is worse than if we just migrate the index
  • But, it does mean we can punt index migration logic until major versions, which does simplify some things

[clintongormley]

I have a proposal which, I think, simplifies the migration process and allows for seamless integration with plugins.

Tracking migrations

Every saved object should include a field which records which plugins have an interest in that object and the version of the plugin that last upgraded the object. For instance:

    "versions": ["core:6.3.0", "my-plugin:6.2.0"]

This way, Core can apply all upgrades to migrate the object from 6.3.0 to the current version, regardless of whether my-plugin is installed or not. Later, when my-plugin is installed, a further migration can be run which can apply its own list of upgrades.

This property should be a required value that every plugin needs to provide, so that any object which is missing this value can be considered to come from the last known version without support for this property.

Core and plugins would be passed all objects, and they would be responsible for determining whether they should make changes to the object or not.

Specifying migrations

I don't see the point of using hashes to identify individual migrations - it overcomplicates things. I'd just have a single migration function per version, and deal with version changes when backporting.

Always reindex

I would always run migrations by reindexing to a new index. On the subject of limiting breaking changes, this could be done in minor versions by first applying the existing mapping to the new index before running any upgrades. On major version upgrades, you wouldn't apply the original mapping but instead build the new mapping entirely from core and plugins.

It's possible that an index has been upgraded to (eg) 7.1.0 but still contains plugin data for an uninstalled plugin that comes from 6.3.0. When installing the plugin, the index would need to be migrated in "major" mode.

Require user interaction

I wouldn't run the migration automatically, at least not at first. I would make the index read-only and lock the Kibana interface saying that a migration is required. When the user presses a button, run the migration and report back, saying that they can delete the old index once they're satisfied that the migration has been successful.

[chrisdavies]

@clintongormley Thanks for the feedback. I like your suggestions.

Every saved object includes a field like "versions": ["core:6.3.0", "my-plugin:6.2.0"]

I like this. I had an implementation a few iterations back that did almost exactly this, except it stored it like so: {core: '6.3.0', 'my-plugin': '6.2.0'} which I think I prefer to the array. Is there a reason to prefer the array approach instead?

The downside to storing this on a per-doc basis is that it makes the "is this index up to date?" check more complex. Is there a reason to store this per doc instead of at the index level?

I don't see the point of using hashes to identify individual migrations

Yeah. I don't like the chekcsum, either, and changed my mind about it after posting my comment-- I forgot to update the issue. The original idea behind a checksum was that it allowed us to enforce migration ordering (this was before we tied migrations to specific Kibana versions) and it allowed for a very fast "is this up to date?" check. But I'm with you.

Always reindex

This is also what we were doing originally, and I also prefer this. It allows us to ensure that any changes made by migrations are accurately reflected in the index while also preventing data-loss from happening. I think it's a big enough win that it's worth doing. @epixa should weigh in on this though, as he has differing opinions.

A challenge with this is if folks enable / disable plugins while doing migrations. It could mean running multiple migrations per Kibana version. The original migration implementation managed this by reindexing to an index that looked something like this: .kibana-{VERSION}-{SHA_OF_PLUGINS} so that there would not be conflicting index names, so that's solvable, but:

• You may end up w/ more indices than you'd like
• Adds friction to the enable / disable plugin scenario

Require user interaction

I like making migrations an explicit step. It hopefully means that problems with migrations will be detected early on, rather than down the road after they've crept into various parts of the system.

I think the main challenge to a UI approach is, who has permission to click the "Migrate This Thang" button?

Anyone? Migrations are not an x-pack feature, so they can't fully rely on auth, though they can optionally rely on it. I think OSS customers would just have to live with the fact that anyone can click this.

For x-pack customers we could restrict clickability to only those users who have full privileges to the .kibana index (assuming we have such a role / concept).

An alternative solution that we have kicked around is:

• If migration is required, put Kibana into a "red" state w/ an appropriate message
• Don't provide a "Migrate" button in the UI, but provide clear operational guidelines on how to run migrations (as a cli, say)
• Operations folks can run migrations via a terminal command or whatever, maybe starting kibana with a --migrate flag or something

This approach has higher friction, but does get around the permissions issue.

[clintongormley]

I like this. I had an implementation a few iterations back that did almost exactly this, except it stored it like so: {core: '6.3.0', 'my-plugin': '6.2.0'} which I think I prefer to the array. Is there a reason to prefer the array approach instead?

Only that you use fewer fields. I'd be ok with the object approach too.

The downside to storing this on a per-doc basis is that it makes the "is this index up to date?" check more complex. Is there a reason to store this per doc instead of at the index level?

The thing I liked was that (after we move to this approach) it provides a mechanism for each plugin to indicate its interest in a particular object, which makes it easy for a plugin to decide whether it needs to do anything or not. It'd also be OK to store the upgrade levels in a doc in Kibana which summarises the status for the index. Or, with your field structure, you could use a max aggregation to figure out whether migrations are required.

A challenge with this is if folks enable / disable plugins while doing migrations. It could mean running multiple migrations per Kibana version. The original migration implementation managed this by reindexing to an index that looked something like this: .kibana-{VERSION}-{SHA_OF_PLUGINS} so that there would not be conflicting index names, so that's solvable, but:

For index names, we don't need the SHA_OF_PLUGINS, we can just have a counter.

• You may end up w/ more indices than you'd like

Easy to provide a UI function to clear up old indices.

• Adds friction to the enable / disable plugin scenario

Disable doesn't need to do anything, just enable. And we could use that max-agg to check if anything needs to be done. Obviously that'll only work once all plugins actually record an interest in saved objects.

For x-pack customers we could restrict clickability to only those users who have full privileges to the .kibana index (assuming we have such a role / concept).

I like the idea of doing it through the UI instead of the CLI. It can run as the Kibana user, so doesn't require the user to have admin permissions (although we could provide that option as a setting). If somebody has installed a new Kibana, then the migration needs to be run regardless of who runs it. The only downside of letting any user run it is that the admin doesn't have control over the timing.

[chrisdavies]

I'm sold.

Whether a plugin needs to do anything or not

I think the migration system can automatically determine this, as plugins register migrations on a type + version basis, and we'll know the type + version of docs in the index and in imported files.

That said, the max aggregation is a good solution, and there are other advantages to storing things on a per doc basis. I'll weigh the pros/cons and make a decision, there.

Obviously that'll only work once all plugins actually record an interest in saved objects.

We can do this now, as plugins currently register migrations on a per type basis, so we'll know if we need to migrate or not.

[chrisdavies]

I have the current direction outlined in the readme in this PR, and have also outlined possible changes per @clintongormley's comments.

@epixa and @ @clintongormley, I think the two of you might want to kick around the conflicting options that have been proposed (see the Outstanding questions section of the readme). I'm going to hold off on moving this forward until that's resolved. Happy to hop on Zoom or similar, if need be.

[epixa]

@chrisdavies Thanks for doing due diligence on the compatibility approach. I'm disappointed it won't work since it was a convenient low-risk stepping stone to migrations, but I can't think of any solution to the two problems you raised that doesn't involve dramatically complicating the whole system.

That does leave us with actually executing migrations.

We keep talking about major versus minor, but migrations need to be possible in patch versions as well. If we introduced some bad bug in a minor that resulted in broken data, we need to be able to release a migration that fixes that problem in a patch version.

Every saved object should include a field which records which plugins have an interest in that object and the version of the plugin that last upgraded the object.

This would mean that things like security and tags will need to upgrade all existing objects in the index as well as add themselves to every new object at write time. I don't necessarily think that's a problem, but this will need to happen in place rather than requiring a full reindex, otherwise simply having security enabled will result in a reindex on every upgrade.

Require user interaction

I think the benefits of making this an active operation don't outweigh the drawbacks. Kibana isn't just a UI, and it's not acceptable for us to block the REST API on a user initiated action like this. As we roll out more and more capabilities at the REST API level, we need to be working toward minimizing downtime on upgrades rather than increasing it.

That aside, we can't simply put things in read-only state and have any expectation that Kibana will behave in a sensible way. Simply loading the Kibana UI requires UI settings which are saved objects, and we can't trust that those are compatible with the current version pre-migration.

It'd also just not be a great experience. Imagine if every time we upgrade demo.elastic.co some random visitor stumbles upon an admin interface asking them to migrate.

Fortunately, with a reindex approach, there shouldn't be any real risk to just running the migration on startup. There'd still be downtime, but it would be as minimal as possible. There's no dealing with a read-only state as we simply don't listen on a port until things are square.

[clintongormley]

This would mean that things like security and tags will need to upgrade all existing objects in the index as well as add themselves to every new object at write time. I don't necessarily think that's a problem, but this will need to happen in place rather than requiring a full reindex, otherwise simply having security enabled will result in a reindex on every upgrade.

@epixa can you explain more about ^^ - I don't understand why anything needs to happen in place? My understanding is that a doc is retrieved from the old index, funnelled through core migrations, then plugin migrations, then written to the new index. So we'd only need to do this when core or a plugin is upgraded.

[epixa]

@clintongormley and I spoke directly about this to get on the same page, and this is where we landed:

1. It does look like we need to support full migrations rather than just a compatibility layer. The only way I can think of to address the concern you raised about compatibility layer in find would be to add a ton of complexity to how transformations are defined to give them field level context, and that's just not a rabbit hole worth going down at this time, if ever.

2. While as a policy we will seek to limit migrations wherever possible, we need to be able to run migrations in any version, including a patch. This is important in the event that a plugin gets installed that requires a migration but also in the event that there exists an egregious bug that results in bad data that we need to address in a patch version.

3. The proposal to tag objects with the last kibana and plugin versions that ran migrations on them seems pretty solid. It also means that things like object import actually rely on the same mechanism for determine what transformations need to happen as the migration system itself, which is nice.

4. Always reindexing is the way to go. It's the only definitive way we can help prevent bad data loss from stupid mistakes that Kibana or plugins make.

5. Migrations should happen at Kibana startup and shouldn't require user interaction. Kibana's more than a UI, and we need to limit the downtime during upgrade as much as possible. There are multiple options for how to make Kibana behave during a migration to make it "nicer" for users, but initially we should just block Kibana from binding to a port until the migration is finished. This way it effectively just looks like a longer startup time for Kibana when a migration runs. We can continue to iterate on this as needed.

@chrisdavies What do you think?

[chrisdavies]

I think it's a solid plan. It's actually pretty close to the original plan, but with some nice simplifications (like tracking migrations based on versioning rather than clunky hashes / migrationState). I agree completely that the consistency is a nice feature of this approach.

So yaya! I'm really happy with this direction.

Thanks, @epixa @clintongormley for hopping on this and hammering it out.

[chrisdavies]

@epixa @tylersmalley

A couple of questions remain that I'm unsure about:

What should the saved object REST API do when it is sent documents that have no migration version information associated with them?

Do we assume the docs are up to date or do we assume they are out of date? It seems that we need to assume they are out of date (e.g. they predate the migrations feature), but:

• If out-of-date, this means Kibana and all future API callers will need to pass accurate version information to the saved object API
  • Is this a realistic demand for us to make?
  • It means API consumers need to have an intimate knowledge of our versioning scheme
  • In the browser, we may be able to do this automatically via the saved object browser client
  • When Kibana loads in the browser, we can fetch a dictionary of plugin -> type -> version
  • When the browser writes a saved object, the saved object client assumes it is up to date (unless it is explicitly told otherwise), and it slaps on the version information of the plugin that owns the document's type prior to sending it to the server
  • This assumes that the browser is properly building the object, but we can build a validation layer into the server at some point, if we wish to verify this prior to writing
  • This'll get more complex if we introduce multi-type documents (e.g. docs that have a primary type, but also anciliary data such as ACLs or whatever)
• If up-to-date, this means we'll just pass them into the index, and hope for the best
  • This seems likely to lead to improperly formed data in the index

What do we do if the saved object API receives an "update" call with a document whose migrationVersion is out of date?

I think we have to fail the update, as it seems unrealistic to ask migration authors to write migrations in such a way that they can properly handle partial documents (and update calls will have partial documents).

EDIT: Given the complexity of expecting everyone to pass our API proper migration version info, it may be better / simpler to say that we don't pass migration version info with every saved object write. Instead, we assume docs w/ no migration version are up to date. And, we tell API consumers that if they are calling the Kibana API, they need to update their code to send us up-to-date documents or the resulting behavior is undefined.

[epixa]

What should the saved object REST API do when it is sent documents that have no migration version information associated with them?

I think the saved object API should only deal in terms of the latest document format, which does mean that we shouldn't introduce BWC breaks on the data level within minors, but that was our plan anyway. This is a decision we can change in the future if we deem it necessary.

The import API is the only API that must be able to deal with older objects, and for that API I think we should assume all objects will contain a version or they are treated as crazy-old (this is the precise way to refer to objects that predate this migration system). Of course we'll need to ensure the export API includes the current version on each object.

What do we do if the saved object API receives an "update" call with a document whose migrationVersion is out of date?

As in my previous answer, I don't think these APIs should deal with migration at all for now. I'd take it a step further and say that we should explicitly deny any request (other than import) that even specifies a migrationVersion property, which will prevent people from doing things they don't expect and will give us the ability to introduce a more robust handling of migrationVersion as a feature in a future minor version if we felt it necessary.

[epixa]

I'm going to close this as saved object migrations were merged in #20243