search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
138 stars 272 forks source link

[RELEASE] Release version 2.17.1 #5046

Closed github-actions[bot] closed 2 weeks ago

github-actions[bot] commented 1 month ago

Release OpenSearch and OpenSearch Dashboards 2.17.1

I noticed that a manifest was automatically created in manifests/2.17.1. Please follow the following checklist to make a release.

How to use this issue

## This Release Issue This issue captures the state of the OpenSearch release, its assignee (Release Manager) is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. For more information check the the [Release Process OpenSearch Guide](https://github.com/opensearch-project/opensearch-build/wiki/Releasing-the-Distribution).

Please refer to the following link for the release version dates: Release Schedule and Maintenance Policy.

Entrance Criteria

Criteria Status Description  Comments
Each component release issue has an assigned owner :green_circle:
Documentation draft PRs are up and in tech review for all component changes :green_circle:
Sanity testing is done for all components :green_circle:
Code coverage has not decreased (all new code has tests) :green_circle:
Release notes are ready and available for all components :green_circle:
Roadmap is up-to-date (information is available to create release highlights) :green_circle:
Release ticket is cut, and there's a forum post announcing the start of the window :green_circle:
Any necessary security reviews are complete :green_circle:

OpenSearch 2.17.1 exit criteria status:

Criteria Status Description  Comments
Performance tests are run, results are posted to the release ticket and there no unexpected regressions :green_circle:
No unpatched vulnerabilities of medium or higher severity that have been publicly known for more than 60 days :green_circle:
Documentation has been fully reviewed and signed off by the documentation community. :green_circle:
All integration tests are passing :green_circle:

OpenSearch-Dashboards 2.17.1 exit criteria status:

Criteria Status Description  Comments
Documentation has been fully reviewed and signed off by the documentation community :green_circle:
No unpatched vulnerabilities of medium or higher severity that have been publicly known for more than 60 days :green_circle:
All integration tests are passing :green_circle:

Preparation

Campaigns

Version Increment - _Ends September 24th 2024

Code Complete - _Ends September 24th 2024

Release Candidate Creation and Testing - _Ends September 25th 2024

Pre Release - _Ends September 30th 2024

Release - _Ends October 1st 2024

Release Checklist.


Release Checklist

### Pre-Release activities - [ ] Promote Repos. - - [ ] OS - - [ ] OSD - [ ] Promote Artifacts. - - [ ] Windows - - [ ] Linux Debian - - [ ] Linux RPM - - [ ] Linux TAR - [ ] Consolidated Release Notes. ### Release activities - [ ] Docker Promotion. - [ ] Release Validation part 1. - - [ ] OpenSearch and OpenSearch Dashboard Validation. - - [ ] Validate the native plugin installation. - [ ] Merge consolidated release notes PR. - [ ] Website and Documentation Changes. - - [ ] Merge staging website PR. - - [ ] Promote the website changes to prod. - - [ ] Add website alert. - [ ] Release Validation part 2. - - [ ] Validate the artifact download URL's and signatures. - [ ] Release Validation part 3. - - [ ] Trigger the validation build (Search for `Completed validation for <>` in the logs). - [ ] Maven Promotion. - [ ] Publish blog posts. - [ ] Advertise on Social Media. - [ ] Post on public slack and Github Release issue.


Post Release

Components

Replace with links to all component tracking issues.

Component On track Release Notes
{COMPONENT_ISSUE_LINK} {INDICATOR} {STATUS}
Legend

| Symbol | Meaning | | -------- | ---------- | | :green_circle: | On track with overall release | | :yellow_circle: | Missed last milestone | | :red_circle: | Missed multiple milestones |

peterzhuamazon commented 3 weeks ago

We have recently found an issue related to OSD, due to the Chrome browser update:

The issue seems coming from chrome itself, with an issue opened in here:

A temp workaround in OSD core has been sent as well:

We want to analyze the situation and see if it grants a potential patch release such as 2.17.1.

cc: @Pallavi-AWS @dblock @anirudha @ashwin-pc @AMoo-Miki @getsaurabh02

Thanks.

getsaurabh02 commented 3 weeks ago

Thanks @peterzhuamazon for adding details. @ashwin-pc Can we also add if there is any workaround from the Chrome side, which users can do and we can share in form of documentation?

I experienced the same issue while using regular browsing, but it worked in private browsing. I cleared my cache and cookies, which resolved the problem.
ashwin-pc commented 3 weeks ago

Thanks for opening the issue @peterzhuamazon @getsaurabh02. To answer your questions:

How many versions of Chrome browser carry this bug?

its started in version 129 and still exists in the latest 130 version

Is there a quick workflow that users can apply, so that we dont need to do a immediate new patch release?1

Users can use an alternate browser Both Edge and Safari dont seem to have this bug. That being said, many of the other browsers are built on top of Chromium, so its very likely that they will also have this bug when they update to use the latest version of Chrome.

What is the impact of this issue?

Discover is unusable since the results table is covered by a red mask

Can we also add if there is any https://github.com/opensearch-project/OpenSearch-Dashboards/issues/8250#issuecomment-2363923878 from the Chrome side, which users can do and we can share in form of documentation?

The work arounds suggested here expect the user to change the CSS on their page using the developer tools panel. Using an alternate browser is an easier workaround though and the suggestions in the issue were most for the code fix we needed to make.

getsaurabh02 commented 3 weeks ago

Thanks @ashwin-pc. One last thing since user here mentioned the below workaround irrespective of the CSS update. Is it not working?

I experienced the same issue while using regular browsing, but it worked in private browsing. I cleared my cache and cookies, which resolved the problem.

Adding maintainers for any feedback/concerns @AMoo-Miki @dbbaughe @msfroh

ashwin-pc commented 3 weeks ago

The user there was calling out that the issue for them was not always reproducible, but their fix/workaround does always work. The issue for me and @AMoo-Miki however was always reproducible.

getsaurabh02 commented 3 weeks ago

Thanks, @ashwin-pc.

I will allow today for any final feedback or concerns from the maintainers. If none are raised, we will start the code freeze tomorrow and proceed to release as soon as the final RC is ready. The release window is typically one week, so we expect to have the final RC ready for release by Tuesday, October 1st.

Please ensure that no other commits (aside from those discussed on this thread) are included in this patch release. Any other changes on the 2.17 branch, apart from those tagged in the patch issue, will be reverted.

Tagging @Divyaasm and @peterzhuamazon for support on this release. We will begin the announcement on the public slack.

getsaurabh02 commented 3 weeks ago

@ashwin-pc. Few more questions, while we prep for the patch release in parallel:

The user there was calling out that the issue for them was not always reproducible, but their fix/workaround does always work. The issue for me and @AMoo-Miki however was always reproducible.

Can we please confirm this, since it will determine the urgency/need of the fix and patch release.

its started in version 129 and still exists in the latest 130 version

Is there any eta on the issue being tracked from the Chrome side? Is it okay to ask user to use other versions of Chrome of other browser, since this issue anyways affects all versions of OSD.

ashwin-pc commented 3 weeks ago

@ashwin-pc. Few more questions, while we prep for the patch release in parallel:

The user there was calling out that the issue for them was not always reproducible, but their fix/workaround does always work. The issue for me and @AMoo-Miki however was always reproducible.

Can we please confirm this, since it will determine the urgency/need of the fix and patch release.

Yes we have confirmed the issue is always reproducable and we have even submitted the exact reproduction steps to the chrome team which they were also able to reproduce it on their end

its started in version 129 and still exists in the latest 130 version

Is there any eta on the issue being tracked from the Chrome side? Is it okay to ask user to use other versions of Chrome of other browser, since this issue anyways affects all versions of OSD.

They havent provided me any ETA yet on the issue. They have acknowledged the issue though. Asking users to use another browser is a valid workaround but given that al major browsers besides Firefox and Safari are built on top of Chrome, this issue will grow in size even if it is patched in Chrome. I dont think we should rely on the browser fix alone to mitigate this issue.

AMoo-Miki commented 3 weeks ago

Is there any eta on the issue being tracked from the Chrome side? Is it okay to ask user to use other versions of Chrome of other browser, since this issue anyways affects all versions of OSD.

@getsaurabh02, according to Chrome's issue tracking system, the bug we reported is marked as a P1 with an S2 level, and is slated for the 131 release. If the fix makes it into their 131 release, it is expected to be available on November 12, 2024. However, if the Chrome team misses the 131 release window, their next opportunity to address the issue will be with the 132 release on January 7, 2025. In the best-case scenario, we are anticipating a minimum of another 50 days before Chrome users will see a resolution to this bug. The next best scenario would involve a wait of 106 days.

eirsep commented 3 weeks ago

https://github.com/opensearch-project/security-analytics/issues/1319 https://github.com/opensearch-project/security-analytics/issues/1258 Users have reported bugs in threat intelligence feature which are critical to correct functioning and causing stuck tasks if they are not fixed

Cc: @getsaurabh02

Divyaasm commented 3 weeks ago

Please review the list of commits added to 2.17 branch after minor release

And kindly include only the commits that are related to CVE, critical bug fixes or backward incompatible fixes (for non-experimental feature) or integ-test-fixes which were already tagged in this issue.

Revert all the commits which are not applicable.

Attaching 2.17 build manifest for commit reference OS manifest OSD manifest

List of commits: OS: https://github.com/opensearch-project/OpenSearch/pull/15935

k-nn: https://github.com/opensearch-project/k-NN/pull/2132 https://github.com/opensearch-project/k-NN/pull/2121 https://github.com/opensearch-project/k-NN/pull/2123

ml-commons: https://github.com/opensearch-project/ml-commons/pull/2966 https://github.com/opensearch-project/ml-commons/pull/2957 https://github.com/opensearch-project/ml-commons/pull/2953 https://github.com/opensearch-project/ml-commons/pull/2947 https://github.com/opensearch-project/ml-commons/pull/2940 https://github.com/opensearch-project/ml-commons/pull/2936

OSD: https://github.com/opensearch-project/OpenSearch-Dashboards/commits/2.17/ - Commits after Sep 13th

opensearch-dashboards-functional-test: https://github.com/opensearch-project/opensearch-dashboards-functional-test/commit/1d64b0410c7421ae4bc1b38da991877646d1d070 https://github.com/opensearch-project/opensearch-dashboards-functional-test/commit/8f8ec2a7e86a618da22bbd616851b4ed100a2a3b

anomaly-detection-dashboards-plugin https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/commit/e8035458fd7624c0c54b94e757a4c4ae3762ac6a - Reverted****

dashboards-notifications: https://github.com/opensearch-project/dashboards-notifications/commits/2.17/ Commits from Sep 17th

alerting-dashboards-plugin: https://github.com/opensearch-project/alerting-dashboards-plugin/commits/2.17/ Commits from Sep 17th

getsaurabh02 commented 3 weeks ago

@eirsep is threat intelligence a new feature and under feature flag? Why these cannot wait until until 2.18 given this was not called out yet? Can you elaborate the impact? Patch releases are only for critical fixes (not important), also exercised on prominent code path. 

https://github.com/opensearch-project/security-analytics/issues/1319
https://github.com/opensearch-project/security-analytics/issues/1258
Zhangxunmt commented 3 weeks ago

The above commits in ml-commons are all bug fixes and tutorial document improvement. They are all needed in 2.17.1.

vamshin commented 3 weeks ago

The above commits in k-NN are all critical performance improvements. These improvements are mainly in the areas of disk optimized vector search, which is a new feature being launched in 2.17.0. These fixes have gone through the perf benchmarking and testing. They are all needed in 2.17.1.

getsaurabh02 commented 3 weeks ago

@Zhangxunmt Thanks for dropping in the comment. Can you please help anwer few of the questions on the fixes to help evaluate the urgency/criticality :

I will like to iterate that this patch release should only contain critical bugs or cve fixes to unblock open-source users. For ex this change doesn't look like a critical fix.

Zhangxunmt commented 3 weeks ago

Yes they are all related to the new feature in 2.17.

Yes, 1 more feature flag to include in this PR. https://github.com/opensearch-project/ml-commons/pull/2982

These are the bugs after the bug bash in 2.17. The bug bash happened only after 2.17 RC was available. After the bugs fixed, there weren't enough time left for 2.17.

These bugs will impact the usefulness of the new feature in 2.17.

They were called out after multiple times before the release. But they were not included in the final RC.

This comes along with the 2.17 features so it's better to have them in 2.17.

shatejas commented 3 weeks ago

https://github.com/opensearch-project/k-NN/pull/2132 https://github.com/opensearch-project/k-NN/pull/2140

Both are performance related 2132 - Customers will not get concurrent segment search if the setting is auto introduced in 2.17 on core 2140 - Customers will see higher latencies due to a refactor done to reduce memory footprint and support disk based ann both of which are introduced in 2.17

Is this feature under a feature flag or explicit setting? 2132 - yes its a concurrent segment search setting 2140 - no

How does this bug surface? Is it the most exercised flow? 2132 - Slower query latencies for customers with auto setting 2140 - yes performance degradation on force merge for all customers

What is the impact of this bug?

Performance impact

Why was this not called out yet?

These were merged pretty close to release and the performance impact was identified as a part of review nightly benchmark runs metrics. The review happens weekly

Why this cannot wait until 2.18?

User will have a bad experience with higher latencies

getsaurabh02 commented 3 weeks ago

@Zhangxunmt @shatejas Anything introduced as a new feature in 2.17 or under a Feature Flag do not qualify as a critical fix for patch release. Critical Fixes are only for changes where an open-source user will need to revert back due to failure or poor experience post upgrade.

Can we please eliminate the ones which are new features or under feature flag, and list those still qualifying as critical.

ryanbogan commented 3 weeks ago

https://github.com/opensearch-project/k-NN/pull/2123

jmazanec15 commented 3 weeks ago

https://github.com/opensearch-project/k-NN/pull/2121

amsiglan commented 3 weeks ago

[Notifications] Fix for broken deny list and update to check for all resolved IPs in the deny list to be included in this patch release - https://github.com/opensearch-project/notifications/pull/964

Divyaasm commented 3 weeks ago

Release Candidate Details for 2.17.1

OpenSearch - Build 10331 OpenSearch Dashboards - Build 7969

Check how to install opensearch and dashboards on different platforms

opensearch-ci-bot commented 3 weeks ago

Core Components CommitID(after 2024-09-27) & Release Notes info

Repo Branch CommitID Commit Date Release Notes Exists
OpenSearch [1893d20797e30110e5877170e44d42275ce5951e] False
OpenSearch-Dashboards [62cc0320399aef63aa09689aaaf000adafbedeef] False
alerting [bb01d2724dbfa4d2851d840203b84c6c4745be85] False
alertingDashboards [d9fe4d06369717fb7e3a3ca2bb20f4751c8b975f] False
anomaly-detection [59ec24d9c3f6033c3f9cdee035e366cecf694224] False
anomalyDetectionDashboards [296b44d811d7b0adbe4ef8218d6f84a2757ac484] False
assistantDashboards [bed2c93e0f9b29f616fc0eb44cecc8eb1f8c5dda] False
asynchronous-search [a31fb4268dd0bec733aef4e1764a8487deb2e053] False
common-utils [80e0945a1e7a7c1ee2725a9dcff1c5d4ac8d46f0] False
cross-cluster-replication [d72750e1e7c3bafc16af217cfa1d52fa83f50f22] False
custom-codecs [49130900d9006e8f00cb4f55d93b0ba9ca3e1a60] False
customImportMapDashboards [8997a363fb6f25892e7f8873eecd1ea5bb53c42f] False
flow-framework [e907cd00eb610e1aa7685ff9887f7a18c0d09904] False
functionalTestDashboards [2.17] False
ganttChartDashboards [c40915b7534466832afad19042d422e347c05c40] False
geospatial [51510fa0a424140562fa795f3430f1af0286aa86] False
index-management [635a535951b683fd9f36d99b927a7cc5d9b30470] False
indexManagementDashboards [3ef3b556eb8b23557aa430e23a1bdeec8aca2aa2] False
job-scheduler [91825ce10c8075dd5bbad5de0c309968d610658e] False
k-NN [5dc38430d24431e9545b01f65d461ebc96d89f6c] False
ml-commons [9e4844f112fdc05c719475aa52b093bed6ae2497] False
mlCommonsDashboards [8a01341cadbeffd1d51e6156189eea1a096ed3e8] False
neural-search [9e529b8b92a967242a50bd3a3c4e41dbf766e4f5] False
notifications [df45059728f4ee0ee62bc1167df08ad4113cd7db] False
notifications-core [df45059728f4ee0ee62bc1167df08ad4113cd7db] False
notificationsDashboards [dedf3851752da6da58ef4357c4a12188315b74e7] False
observabilityDashboards [62cc940a727134cca3f2fa35ca232fef1be9c5e4] False
opensearch-observability [552e04e9064806b6a8b77a24faf2f97e1f8fd328] False
opensearch-reports [5af0de7a195d08ddf9410db0d2bb3b9d2cf4657d] False
opensearch-system-templates [3ffa8567a980d6a9f1f2c37d6ccf4d47804dd29b] False
performance-analyzer [759d2ef6604625e1f41dc0eb6d27dbbbfa8ccda8] False
query-insights [7183618210865b69af20e817ec899085febeb7ba] False
queryWorkbenchDashboards [b92d5cec8be1ecc75bb0285d530910a2f87a0b82] False
reportsDashboards [41815cb16ee41cad43b68497e1373342c473ae39] False
searchRelevanceDashboards [29225dbb6b7024596ec13880796eb4516a157096] False
security [d47df0642f4a8db7617adfdcd2cac539dd3dbae5] False
security-analytics [d7e9295503f1106047ba4cf126c5998a7eda2cb2] False
securityAnalyticsDashboards [7d22d22d8714bea0576649289acfc6b0cd9bc121] False
securityDashboards [6bc2e342da90a938956289f6bc36c6313a9a07b7] False
skills [e526edea4cb168e475684242e0b261c1d57abce4] False
sql [fe219794aecd57cd9c8aa4711d676302a14893a8] False
Divyaasm commented 3 weeks ago

Latest Release Candidate Details for 2.17.1

OpenSearch - Build 10332 OpenSearch Dashboards - Build 7969

Check how to install opensearch and dashboards on different platforms

Divyaasm commented 3 weeks ago

Integration Test Results

Thank you

Vikasht34 commented 3 weeks ago

opensearch-project/k-NN/2152 Is this related to new feature introduced in 2.17? Yes Is this feature under a feature flag or explicit setting? No How does this bug surface? Is it the most exercised flow? Yes , if user does not provide higher rescoring factor then it will be defaulted to lower rescoring factor which gives low re-c all which is not a good user experince. What is the impact of this bug? The user will not get good re-call on semantics search Why was this not called out yet? This was merged close to the release deadline and was supposed to be included pre-release. Why this cannot wait until 2.18? User can run into performance issues. This will give them a bad experience with the feature that we released.

Divyaasm commented 3 weeks ago

Integration Test Results

Thank you

Divyaasm commented 2 weeks ago

Native Plugin Validation


-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission config#plus read
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission aws.configFile read,write
* java.util.PropertyPermission aws.sharedCredentialsFile read,write
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
* java.util.PropertyPermission opensearch.path.conf read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3 with folder name repository-s3```
Divyaasm commented 2 weeks ago

Signature validation


gpg --verify opensearch-2.17.1-linux-x64.tar.gz.sig opensearch-2.17.1-linux-x64.tar.gz
gpg: Signature made Tue Oct  1 13:39:56 2024 PDT
gpg:                using RSA key C2EE2AF6542C03B4
gpg: Good signature from "OpenSearch project <opensearch@amazon.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: C5B7 4989 65EF D1C2 924B  A9D5 39D3 1987 9310 D3FC
     Subkey fingerprint: 2187 3199 B103 0FCD 49DA  83F8 C2EE 2AF6 542C 03B4
Divyaasm commented 2 weeks ago

OpenSearch 2.17.1 has been released to the public

dbwiddis commented 4 days ago

@Divyaasm see https://github.com/opensearch-project/OpenSearch/issues/16350