Open joshpalis opened 1 year ago
joshpalis
commented
1 year ago @prudhvigodithi @nknize @dblock @vibrantvarun @praveensameneni @sean-zheng-amazon @anirudha @bbarani @gaiksaya @peterzhuamazon
For the dependent plugin owners, please let us know if you have any questions / concerns regarding items [1] and [2]. We shall move forward with raising these PRs starting December 5th.
praveensameneni
commented
1 year ago LGTM
minalsha
commented
1 year ago @sean-zheng-amazon @anirudha can i please get your signoff here?
sean-zheng-amazon
commented
1 year ago I think this is a great move, long waiting for. Just one thing to confirm, the gradle change is the only change needed from plugin side right? and also assumed this shouldn't cause any backward compatibility issue.
joshpalis
commented
1 year ago Hi @sean-zheng-amazon, yes the aforementioned gradle changes are the only modifications needed from the plugin side. As for any backward compatibility issues that may come up (if any), we shall also take care of resolving.
prudhvigodithi
commented
1 year ago Hey @saratvemulapalli @joshpalis quick question from the comment, what about for the users who do upgrade/rolling-upgrade from 1.x to 2.x? In this case 1.x the job scheduler is pre-installed with the bundle, but for 2.x (where job-scheduler is now a native plugin) the job scheduler should be explicitly installed, how do we plan to handle this? Can this be a red flag and break anything in the running cluster? Also Upgrade from one major/minor to next major/minor version where job-scheduler is now a native plugin would also fall under same scenario. @bbarani @peterzhuamazon @dblock @CEHENKLE
rupal-bqcurl -XGET https://localhost:9200/_cat/plugins -ku 'admin:admin'
039cd4094188 opensearch-alerting 2.4.0.0
039cd4094188 opensearch-anomaly-detection 2.4.0.0
039cd4094188 opensearch-asynchronous-search 2.4.0.0
039cd4094188 opensearch-cross-cluster-replication 2.4.0.0
039cd4094188 opensearch-geospatial 2.4.0.0
039cd4094188 opensearch-index-management 2.4.0.0
039cd4094188 opensearch-job-scheduler 2.4.0.0
039cd4094188 opensearch-knn 2.4.0.0
039cd4094188 opensearch-ml 2.4.0.0
039cd4094188 opensearch-neural-search 2.4.0.0
039cd4094188 opensearch-notifications 2.4.0.0
039cd4094188 opensearch-notifications-core 2.4.0.0
039cd4094188 opensearch-observability 2.4.0.0
039cd4094188 opensearch-performance-analyzer 2.4.0.0
039cd4094188 opensearch-reports-scheduler 2.4.0.0
039cd4094188 opensearch-security 2.4.0.0
039cd4094188 opensearch-security-analytics 2.4.0.0
039cd4094188 opensearch-sql 2.4.0.0
saratvemulapalli
commented
1 year ago @prudhvigodithi it really doesnt matter how the plugin is installed (i.e native vs non-native), after the installation the code is the same from OpenSearch point of view. So rolling upgrade will work as it was before. Customers using non native JS with OpenSearch 1.x should be able to upgrade to native JS with OpenSearch 2.x with no problems. The only difference between native and non-native plugins is:
prudhvigodithi
commented
1 year ago Hey @saratvemulapalli thanks for the info, what I was curious as part of the rolling upgrade is, since the native plugins are default not installed in the bundle, how do we plan to handle this? Do we inform to re-install the plugin manually after the upgrade is done? As the old version would have the job-scheduler installed (that comes out of the box with the bundle) but not when its a native plugin.
minalsha
commented
1 year ago Hi @prudhvigodithi one clarification: If JS is part of core/modules folder, then it is installed out of the box and it becomes a mandated install for customers who may not even use it. We are proposing to move Job Scheduler as a native plugin so that it is part of the Core/Plugins folder since we do not want it to be a mandated installation and customers can determine if they want to install it or not. (All the plugins in the Plugins folder are optional and can be installed optionally by a user). @saratvemulapalli / @joshpalis please confirm if I am mistaken. Thanks
joshpalis
commented
1 year ago @minalsha You are correct, native plugins are optionally installed. @prudhvigodithi is referring to the full bundle of OpenSearch, which installs all the components listed in the input build manifest, which currently includes Job Scheduler. While native plugins are part of the OpenSearch component, these are not installed by default but their artifacts are built and published as part of build/assemble. Once Job Scheduler is relocated to native plugins, since it is a dependency of other component plugins, it is necessary for Job Scheduler to be pre-installed, despite it being a native plugin. This issue is being addressed here : https://github.com/opensearch-project/opensearch-build/issues/2849
saratvemulapalli
commented
1 year ago Hey @saratvemulapalli thanks for the info, what I was curious as part of the rolling upgrade is, since the native plugins are default not installed in the bundle, how do we plan to handle this? Do we inform to re-install the plugin manually after the upgrade is done? As the old version would have the job-scheduler installed (that comes out of the box with the bundle) but not when its a native plugin.
@prudhvigodithi our distributions will have JS installed (doesn't matter if its installed via native/non-native), because there are other dependent plugins. i.e we should install JS within the bundle, if not we cannot ship AD, ISM etc. So, rolling upgrade won't be any different. OpenSearch min distribution will not have JS installed by default, if customers need it they could install it.
joshpalis
commented
1 year ago In order to ensure that relocating Job Scheduler to native plugins does not affect rolling upgrades we will perform rolling upgrades for 2.x -> 3.x and 1.x -> 2.x. Each experiment will first use the original cluster to start an Anomaly Detector and upon completing the rolling upgrade, we will utilize the Get Detector API to retrieve real time analysis information for this particular detector. By retrieving real time analysis information after executing the rolling upgrade, we can confirm that Job Scheduler is still working as a native plugin
2.x cluster configuration:
2.x Logs
3.x cluster configuration:
3.x Logs
Result : Success
Creating Anomaly Detector (ID : J7N6CIUB02zLoZpx4Nd8 )
curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":1,"unit":"Minutes"}},"window_delay":{"period":{"interval":1,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
"_id" : "J7N6CIUB02zLoZpx4Nd8",
"_version" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "test-detector",
"description" : "Test detector",
"time_field" : "timestamp",
"indices" : [
"server_log*"
],
"filter_query" : {
"bool" : {
"filter" : [
{
"range" : {
"value" : {
"from" : 1,
"to" : null,
"include_lower" : false,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 1,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 1,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "JrN6CIUB02zLoZpx3ddb",
"feature_name" : "test",
"feature_enabled" : true,
"aggregation_query" : {
"test" : {
"sum" : {
"field" : "value"
}
}
}
}
],
"last_update_time" : 1670884548730,
"detector_type" : "SINGLE_ENTITY",
"result_index" : "opensearch-ad-plugin-result-test"
},
"_primary_term" : 1
}Starting Detector (ID : J7N6CIUB02zLoZpx4Nd8)
curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/J7N6CIUB02zLoZpx4Nd8/_start"
{"_id":"J7N6CIUB02zLoZpx4Nd8","_version":1,"_seq_no":0,"_primary_term":1}%Retrieving Real Time Analysis Information for Detector (ID : J7N6CIUB02zLoZpx4Nd8) post upgrade
curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/J7N6CIUB02zLoZpx4Nd8?job=true"
{
"_id": "J7N6CIUB02zLoZpx4Nd8",
"_version": 1,
"_primary_term": 1,
"_seq_no": 0,
"anomaly_detector": {
"name": "test-detector",
"description": "Test detector",
"time_field": "timestamp",
"indices": [
"server_log*"
],
"filter_query": {
"bool": {
"filter": [
{
"range": {
"value": {
"from": 1,
"to": null,
"include_lower": false,
"include_upper": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"detection_interval": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"window_delay": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"shingle_size": 8,
"schema_version": 0,
"feature_attributes": [
{
"feature_id": "JrN6CIUB02zLoZpx3ddb",
"feature_name": "test",
"feature_enabled": true,
"aggregation_query": {
"test": {
"sum": {
"field": "value"
}
}
}
}
],
"last_update_time": 1670884548730,
"detector_type": "SINGLE_ENTITY",
"result_index": "opensearch-ad-plugin-result-test"
},
"anomaly_detector_job": {
"name": "J7N6CIUB02zLoZpx4Nd8",
"schedule": {
"interval": {
"start_time": 1670884573449,
"period": 1,
"unit": "Minutes"
}
},
"window_delay": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"enabled": true,
"enabled_time": 1670884573449,
"last_update_time": 1670884573449,
"lock_duration_seconds": 60,
"result_index": "opensearch-ad-plugin-result-test"
}
}1.x cluster configuration:
1.x Logs
2.x cluster configuration:
2.x Logs
Result : Success
Creating Anomaly Detector (ID : oBmvDYUB0LodPD1ay_uK )
curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":1,"unit":"Minutes"}},"window_delay":{"period":{"interval":1,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
"_id" : "oBmvDYUB0LodPD1ay_uK",
"_version" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "test-detector",
"description" : "Test detector",
"time_field" : "timestamp",
"indices" : [
"server_log*"
],
"filter_query" : {
"bool" : {
"filter" : [
{
"range" : {
"value" : {
"from" : 1,
"to" : null,
"include_lower" : false,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 1,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 1,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "nxmvDYUB0LodPD1ayPuC",
"feature_name" : "test",
"feature_enabled" : true,
"aggregation_query" : {
"test" : {
"sum" : {
"field" : "value"
}
}
}
}
],
"last_update_time" : 1670971902856,
"detector_type" : "SINGLE_ENTITY",
"result_index" : "opensearch-ad-plugin-result-test"
},
"_primary_term" : 1
}Starting Detector (ID : oBmvDYUB0LodPD1ay_uK )
curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/oBmvDYUB0LodPD1ay_uK/_start"
{"_id":"oBmvDYUB0LodPD1ay_uK","_version":1,"_seq_no":0,"_primary_term":1}% Retrieving Real Time Analysis Information for Detector (ID : oBmvDYUB0LodPD1ay_uK) post upgrade
curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/oBmvDYUB0LodPD1ay_uK"
{
"_id": "oBmvDYUB0LodPD1ay_uK",
"_version": 1,
"_primary_term": 1,
"_seq_no": 0,
"anomaly_detector": {
"name": "test-detector",
"description": "Test detector",
"time_field": "timestamp",
"indices": [
"server_log*"
],
"filter_query": {
"bool": {
"filter": [
{
"range": {
"value": {
"from": 1,
"to": null,
"include_lower": false,
"include_upper": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"detection_interval": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"window_delay": {
"period": {
"interval": 1,
"unit": "Minutes"
}
},
"shingle_size": 8,
"schema_version": 0,
"feature_attributes": [
{
"feature_id": "nxmvDYUB0LodPD1ayPuC",
"feature_name": "test",
"feature_enabled": true,
"aggregation_query": {
"test": {
"sum": {
"field": "value"
}
}
}
}
],
"last_update_time": 1670971902856,
"detector_type": "SINGLE_ENTITY",
"result_index": "opensearch-ad-plugin-result-test"
}
}In order to ensure that relocating Job Scheduler to native plugins does not affect restart upgrades we will perform restart upgrades for 1.x -> 2.x and 2.x -> 3.x. Similarly, each experiment will first use the original cluster to start an Anomaly Detector and upon completing the restart upgrade, we will utilize the Get Detector API to retrieve real time analysis information for this particular detector. By retrieving real time analysis information after executing the rolling upgrade, we can confirm that Job Scheduler is still working as a native plugin
1.x cluster configuration:
1.x Log
2.x cluster configuration:
2.x Log
Result : Success
Creating Anomaly Detector (ID : NwnzDYUBxGSlphqwAlxq )
curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":10,"unit":"Minutes"}},"window_delay":{"period":{"interval":10,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
"_id" : "NwnzDYUBxGSlphqwAlxq",
"_version" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "test-detector",
"description" : "Test detector",
"time_field" : "timestamp",
"indices" : [
"server_log*"
],
"filter_query" : {
"bool" : {
"filter" : [
{
"range" : {
"value" : {
"from" : 1,
"to" : null,
"include_lower" : false,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "NgnzDYUBxGSlphqwAVwu",
"feature_name" : "test",
"feature_enabled" : true,
"aggregation_query" : {
"test" : {
"sum" : {
"field" : "value"
}
}
}
}
],
"last_update_time" : 1670976307817,
"detector_type" : "SINGLE_ENTITY",
"result_index" : "opensearch-ad-plugin-result-test"
},
"_primary_term" : 1
}Starting Detector (ID : NwnzDYUBxGSlphqwAlxq )
curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/NwnzDYUBxGSlphqwAlxq/_start"
{"_id":"NwnzDYUBxGSlphqwAlxq","_version":1,"_seq_no":0,"_primary_term":1}% Retrieving Real Time Analysis Information for Detector (ID : NwnzDYUBxGSlphqwAlxq) post upgrade
curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/NwnzDYUBxGSlphqwAlxq?pretty"
{
"_id" : "NwnzDYUBxGSlphqwAlxq",
"_version" : 1,
"_primary_term" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "test-detector",
"description" : "Test detector",
"time_field" : "timestamp",
"indices" : [
"server_log*"
],
"filter_query" : {
"bool" : {
"filter" : [
{
"range" : {
"value" : {
"from" : 1,
"to" : null,
"include_lower" : false,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "NgnzDYUBxGSlphqwAVwu",
"feature_name" : "test",
"feature_enabled" : true,
"aggregation_query" : {
"test" : {
"sum" : {
"field" : "value"
}
}
}
}
],
"last_update_time" : 1670976307817,
"detector_type" : "SINGLE_ENTITY",
"result_index" : "opensearch-ad-plugin-result-test"
}
}2.x cluster configuration:
2.x Log
3.x cluster configuration:
3.x Log
Result : Success
Creating Anomaly Detector (ID : 5LoBDoUBG1y14YzvlquP)
curl -XPOST "localhost:9200/_plugins/_anomaly_detection/detectors?pretty" -H "Content-Type:application/json" --data '{"name":"test-detector","description":"Test detector","time_field":"timestamp","indices":["server_log*"],"feature_attributes":[{"feature_name":"test","feature_enabled":true,"aggregation_query":{"test":{"sum":{"field":"value"}}}}],"filter_query":{"bool":{"filter":[{"range":{"value":{"gt":1}}}],"adjust_pure_negative":true,"boost":1}},"detection_interval":{"period":{"interval":10,"unit":"Minutes"}},"window_delay":{"period":{"interval":10,"unit":"Minutes"}},"result_index":"opensearch-ad-plugin-result-test"}'
{
"_id" : "5LoBDoUBG1y14YzvlquP",
"_version" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "test-detector",
"description" : "Test detector",
"time_field" : "timestamp",
"indices" : [
"server_log*"
],
"filter_query" : {
"bool" : {
"filter" : [
{
"range" : {
"value" : {
"from" : 1,
"to" : null,
"include_lower" : false,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "47oBDoUBG1y14YzvlatY",
"feature_name" : "test",
"feature_enabled" : true,
"aggregation_query" : {
"test" : {
"sum" : {
"field" : "value"
}
}
}
}
],
"last_update_time" : 1670977263246,
"detector_type" : "SINGLE_ENTITY",
"result_index" : "opensearch-ad-plugin-result-test"
},
"_primary_term" : 1
}Starting Detector (ID : 5LoBDoUBG1y14YzvlquP)
curl -X POST "localhost:9200/_plugins/_anomaly_detection/detectors/5LoBDoUBG1y14YzvlquP/_start"
{"_id":"5LoBDoUBG1y14YzvlquP","_version":1,"_seq_no":0,"_primary_term":1}Retrieving Real Time Analysis Information for Detector (ID : 5LoBDoUBG1y14YzvlquP) post upgrade
curl -X GET "localhost:9200/_plugins/_anomaly_detection/detectors/5LoBDoUBG1y14YzvlquP?pretty"
{
"_id" : "5LoBDoUBG1y14YzvlquP",
"_version" : 1,
"_primary_term" : 1,
"_seq_no" : 0,
"anomaly_detector" : {
"name" : "test-detector",
"description" : "Test detector",
"time_field" : "timestamp",
"indices" : [
"server_log*"
],
"filter_query" : {
"bool" : {
"filter" : [
{
"range" : {
"value" : {
"from" : 1,
"to" : null,
"include_lower" : false,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
},
"detection_interval" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"window_delay" : {
"period" : {
"interval" : 10,
"unit" : "Minutes"
}
},
"shingle_size" : 8,
"schema_version" : 0,
"feature_attributes" : [
{
"feature_id" : "47oBDoUBG1y14YzvlatY",
"feature_name" : "test",
"feature_enabled" : true,
"aggregation_query" : {
"test" : {
"sum" : {
"field" : "value"
}
}
}
}
],
"last_update_time" : 1670977263246,
"detector_type" : "SINGLE_ENTITY",
"result_index" : "opensearch-ad-plugin-result-test"
}
}
penghuo
commented
1 year ago signoff for observability plugin. because of
joshpalis
commented
1 year ago @penghuo @sean-zheng-amazon @praveensameneni @rupal-bq @anirudha . Please note that due to this change, while we will no longer require modifications to your respective plugin build.gradles for 2.x, we shall be raising PRs to update the maven coordinates for 3.x.
saratvemulapalli
commented
1 year ago @joshpalis dropping 2.5 label and adding in 2.6 as we are shooting for that train.
bbarani
commented
1 year ago @saratvemulapalli We should tag this issue to 3.x based on the latest updates as mentioned here?
joshpalis
commented
1 year ago An update regarding the effort relocate Job Scheduler into core as a native plugin.
Moving Job Scheduler into core has been determined to be a breaking change (modifies the location in which job scheduler artifacts are published to), thus, in order to adhere to semver, Job Scheduler will not be moved into native plugins until 3.0.0.
It is necessary to modify the assemble workflow for the full-bundle OpenSearch distribution, and there is an ongoing effort to determine how the pre-installation of native plugins will be supported. This discussion can be found here : https://github.com/opensearch-project/opensearch-build/issues/2849
Based on this discussion, Job Scheduler will be relocated to core as a native plugin. This issue shall be used to track the associated work items and discuss any questions / concerns.
Draft PR to relocate Job Scheduler to native plugins : https://github.com/opensearch-project/OpenSearch/pull/5453
Update :
Previously, this change has been slated for the 2.6.0 release now that we have ensured that rolling/restart upgrades are not detrimentally affected by this relocation. However, upon consolidating the necessary
build.gradlemodifications for dependent plugins [3], the move to relocate Job Scheduler to native plugins has been determined to be a breaking change. In our efforts to adhere to semantic versioning, we shall relocate Job Scheduler to OpenSearch core as a native plugin on the3.xbranch of OpenSearch, and in order to backport this change to2.x, we shall forgo any dependent pluginbuild.gradlemodifications for2.xand instead publish Job Scheduler artifacts to two separate Maven coordinates.[1] Native Job Scheduler Maven Coordinates :
org.opensearch.plugin:opensearch-job-scheduler-(3 digit version number)org.opensearch.plugin:opensearch-job-scheduler-spi-(3 digit version number)[2] Original Job Scheduler Maven Coordinates :
org.opensearch:opensearch-job-scheduler-(4 digit version number)org.orpensearch:opensearch-job-scheduler-spi-(4 digit version number)The original Job Scheduler Repository shall consume JS Native plugin artifacts[1] and repackage and publish these artifacts to the original Job Scheduler Maven coordinates[2]. By doing so, we can achieve both the relocation of Job Scheduler into native plugins, whilst adhering to semantic versioning by circumnavigating any modifications to how dependent plugins consume Job Scheduler for
2.x.Build and Assemble Workflow Modifications
Update Now that the Job Scheduler relocation to native plugins will target the 3.0.0 release, modifications for the input manifest yml schema will now only be necessary for 3.0.0 and onwards.
Since Job Scheduler acts as a dependency for other plugins (AD, IM, Reporting), relocating this project within core as a native plugin will necessitate changes to the yml schema of the input manifest used to assemble the full distribution of OpenSearch.
Discussions for the input manifest changes has been started here .
Dependent Plugin Modifications Now that the modifications for dependent plugin
build.gradles[3] has been determined to be a breaking change, moving forward, dependent plugins will have no modifications for 2.x in order to adhere to semantic versioning, however this change will be raised for3.x.Simply put, for
2.x, dependent plugins will still consume Job Scheduler artifacts from the original maven coordinates[2], for3.x, dependent plugins will consume Job Scheduler artifacts from the native JS maven coordinates[1].org.opensearch:opensearch-job-scheduler-spi:${job_scheduler_version}. Upon relocation, artifacts will be uploaded as part of the OpenSearch component with the maven coordinatesorg.opensearch.plugin:opensearch-job-scheduler-spi:${job_scheduler_version}Plans to support previous versions of Job Scheduler In order to support security patches to previous versions of Job Scheduler, upon relocation, the original Job Scheduler repository will not be archived.