When create a datasource, it does not have migrationVersion causes the migration cannot be triggered.
Currently, the datasource does not have migration function.(reference) When creating the datasource, the migrationVersion will not be created and this variable will be marked as undefined.
So with the below migration logic, the migration result will depend on doc.migrationVersion. If doc.migrationVersion is undefined, then it will always be considered to up to date and does not need to apply migration for these data.
/**
* Creates a function which migrates and validates any document that is passed to it.
*/
function buildDocumentTransform({
migrations,
}: {
opensearchDashboardsVersion: string;
migrations: ActiveMigrations;
}): TransformFn {
return function transformAndValidate(doc: SavedObjectUnsanitizedDoc) {
const result = doc.migrationVersion
? applyMigrations(doc, migrations)
: markAsUpToDate(doc, migrations);
// In order to keep tests a bit more stable, we won't
// tack on an empy migrationVersion to docs that have
// no migrations defined.
if (_.isEmpty(result.migrationVersion)) {
delete result.migrationVersion;
}
return result;
};
}
Potential Soluion
Find the root cause and fix it in OSD core.
Pros: will permanently resolve the issue
Cons:
Potential risk is high: the document migrator logic is pretty complicated with long history, it is the common used document migrator for all types of save objects. We may need an extra 4 to 5 weeks for the investigation, design and code fix.
Time consuming
[Preferred] Find the workaround which create a dummy migration function for the datasource to ensure the migrationVersion is not undefined when creating a datasource.
Pros:
easy to implement, we can directly move forward to next step
Cons:
Will not fix the issue from the root.
Design
For now, I will use approach 2 to create a dummy migration function for giving the datasource a migrationVersion. And we need to think about two design related questions:
Q1: which version will we use for the dummy migration function
A: I will use 2.4.0 as the migrationVersion since this is the version we introduced the datasource
Add a migrations configuration that contains migration functions keyed by version. The '2.4.0' migration uses the flow function from lodash to apply the migrateDataSource function, indicating this migration should be applied when upgrading to version 2.4.0
Test for the changes
Start the OSD locally and create a datasource (without migration function)
Find the datasource id and then call GET API to check the datasource information
GET API request:
curl -X GET 'http://localhost:5601/api/saved_objects/data-source/d31cd080-da94-11ee-9155-b95eb77f43e8‘curl -X GET 'http://localhost:5601/api/saved_objects/data-source/d31cd080-da94-11ee-9155-b95eb77f43e8‘
Background
When create a datasource, it does not have migrationVersion causes the migration cannot be triggered.
Currently, the datasource does not have migration function.(reference) When creating the datasource, the migrationVersion will not be created and this variable will be marked as
undefined
. So with the below migration logic, the migration result will depend ondoc.migrationVersion
. If doc.migrationVersion is undefined, then it will always be considered to up to date and does not need to apply migration for these data.Potential Soluion
Design
For now, I will use approach 2 to create a dummy migration function for giving the datasource a migrationVersion. And we need to think about two design related questions:
Q1: which version will we use for the dummy migration function
A: I will use 2.4.0 as the migrationVersion since this is the version we introduced the datasource
Q2: how do we implement the migration function:
A:
migrateDataSource
. This is a dummy migration function that returns the document without any changes since we currently do not have any schema change.'2.4.0'
migration uses theflow
function from lodash to apply themigrateDataSource
function, indicating this migration should be applied when upgrading to version 2.4.0Test for the changes
GET API request:
Output is:
We can see there is no migrationVersion included in the output.
We can see the migrationVersion is successfully added.