search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.8k stars 1.82k forks source link

[BUG] OpenSearch version reported of core only #1111

Open stockholmux opened 3 years ago

stockholmux commented 3 years ago

Describe the bug When a user connects to https://<ip-of-cluster>:9200/ the response JSON reports version.number for the core (e.g. 1.0.0). Looking at the plans for 1.0.1 release, it looks like only particular bundled plugins will be updated and not the core. This means if a users updates to version 1.0.1, https://<ip-of-cluster>:9200/ will still respond with 1.0.0 in version.number. This could cause confusion after updates or if a user really needs to distinguish between versions.

To Reproduce Steps to reproduce the behavior:

  1. Go to curl https://localhost:9200/ -u admin:admin -k
  2. See the response - something like: 
    {
"name" : "opensearch-node1",
"cluster_name" : "opensearch-cluster",
"cluster_uuid" : "8oUPWeGFSTOH5cwKaBfuwA",
"version" : {
"distribution" : "opensearch",
"number" : "1.0.0",
"build_type" : "tar",
"build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
"build_date" : "2021-07-02T23:22:21.383695Z",
"build_snapshot" : false,
"lucene_version" : "8.8.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Expected behaviour The user should be able to see the bundled version (as well as the core version) when hitting https://<ip-of-cluster>:9200/

Perhaps something like this:

{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "8oUPWeGFSTOH5cwKaBfuwA",
  "version" : {
    "bundle" : "1.0.1"
    "distribution" : "opensearch",
    "number" : "1.0.0",
    "build_type" : "tar",
    "build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date" : "2021-07-02T23:22:21.383695Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

(see version.bundle)

Although, I could see a different endpoint for the bundle version...

Plugins All OpenSearch

nknize commented 3 years ago

I think this should be an enhancement instead of a bug?

Calling the :9200 endpoint only hits the transport layer on the core to return the MainResponse.

To get installed plugin versions use GET /_cat/plugins.

The "bundle" will be a collection of plugins along with core that will all be different versions. The only way to know the "bundle version" is to maintain a version mapping somewhere that defines, for example, version 1.0.1 as core 1.0.0, security 1.0.1, knn 1.0.0.

If we want a way to report "bundle version" we'll either need to maintain a resource or manifest in every plugin that contains the bundle definition/version number, or define OpenSearch versioning to keep Major.minor consistent across all plugin repo so that they report the same Major.minor version in /_cat/plugins. Since "bundle" is really the aggregate of all repos, I see no other way to achieve this.

stockholmux commented 3 years ago

I think this should be an enhancement instead of a bug?

It's kinda between a bug and an enhancement. IMHO, this is a miss from 1.0.0. There is something fundamentally weird about the situation of someone downloading & installing a new version and having no way to check if it was actually upgraded without looking at the files that are installed. I think it will cause some serious confusion.

nknize commented 3 years ago

There is something fundamentally weird about the situation of someone downloading & installing a new version and having no way to check if it was actually upgraded

The :9200 MainResponse is really the only way to check cluster successfully upgraded....as long as it's reporting 1.*. cat/_plugins will verify the plugins upgraded.

There's no concept of a "bundle" and I'm not sure the current versioning plan supports one in the future.

stockholmux commented 3 years ago

I don't believe the version numbers are outwardly reported for the plugins, at least it wasn't in the 1.0.0 release notes.

Put yourself in the users shoes: Download 1.0.1, check :9200, and it still says 1.0.0. Do you really want a user to have to both know to check cat/_plugins and follow through and cross reference the versions of the plugins vs some webpage? That seems both onerous and hard to automate.

I see this as a mismatch between the ES versioning style (lock-step versioning - e.g. 7.9.0 ES means everything else is 7.9.0 as well) and the way OpenSearch is moving forward with semver and potentially differential versioning. The way ES did it, the version reported by :9200 makes sense. But OpenSearch, it makes way less sense.

nknize commented 3 years ago

We're saying the same thing. The framework to support this issue isn't there (see elastic stack releases where all product versions are coordinated), so it would be an enhancement instead of a bug.

I see this as a mismatch between the ES versioning style (lock-step versioning - e.g. 7.9.0 ES means everything else is 7.9.0 as well) and the way OpenSearch is moving forward

Exactly my point. Currently there is no concept of a "bundle" within the inherited versioning mechanisms. Furthermore, the OpenSearch plan to decouple versions across the repos / plugins does not support the concept of a bundle either without a "many to one" mechanism that maps decoupled versions to a single bundled version. That's why I'm saying the current OpenSearch versioning plan doesn't support the idea of a "bundle" either.

nknize commented 3 years ago

To support this OpenSearch would need a new mechanism; like a package manager or something. e.g., something similar to npm where a package.json defines the "bundle" version with the many bundled plugin dependencies.

Something like this (take it with a grain of salt, just used to illustrate my point / idea):

{
  "name": "opensearch bundle",
  "description": "OpenSearch is a flexible search and analytics platform based on many different components and drivers... the Operating System Kernel of search!",
  "version": "2.0.0",
  "branch": "master",
  "build": {
    "number": 5342,
    "sha": "2cb7eec4e25ffca1a4334ee5b34d5ef2139271da"
  },
  "homepage": "https://www.opensearch.org",
  "bugs": {
    "url": "https://github.com/opensearch-project/opensearch/issues"
  },
  "author": "OpenSearch Project Organization <dev@opensearch.org>",
  "repository": {
    "type": "git",
    "url": "https://github.com/opensearch-project/opensearch.git"
  },
  "resolutions": {
    "**/anomaly-detection": "2.3.4.0",
    "**/security": "^2.1.2.0",
    "**/alerting": "^2.2.2.0",
    "**/k-NN": "^2.1.1.2",
    "**/notifications": "2.3.5.0",
    "**/index-management": "^2.3.2.2",
    ...
  },

In this manner the "bundle" is defined as version 2.0.0 but the plugins may be on higher minor versions by the time the 2.0.0 bundle is ready; ergo, version decoupling. This would also provide the mechanism by which an API can provide a single bundle version w/o having to query :9200 or cat/_plugins to find out all the disparate versions.

stockholmux commented 3 years ago

Gotcha. Sorry for misunderstanding your comment @nknize. My misread. I guess the line between a bug and an enhancement to me is "would the average user be surprised by a behaviour of the software." It's a side concern how this is classified, I suppose.

I do think OpenSearch needs a package manager (via opensearch-cli maybe) although that might be a much larger architectural issue that would probably need to come at a major version. Is there another solution that would allow OpenSearch to become bundle (better terminology: 'distribution') aware and return the expected version?

dblock commented 3 years ago

@nknize We have a mechanism in 1.1 to describe the bundle, a bundle manifest that looks like this. It's going to be dropped along with the rest of the world in the bundle and we could read from that.

So we have two choices:

  1. In a bundle installation, change version.number=bundle version, build_type=tar, build_hash=some kind of bundle hash, build_date=the date of the bundle build from the bundle JSON.
  2. Return the bundle versioning data somewhere else if present as part of this or another API. Where?

(1), from the machine point of view this, at first, sounded like a bad idea, because you could get two different versions for the same -min engine. But bundle closely follows -min in versioning, so semantically it would still be correct. So feels like not a big deal, to me.

WDYT?

nknize commented 3 years ago

I guess the line between a bug and an enhancement to me is "would the average user be surprised by a behaviour of the software."

I agree with this. If a user has a bad time it's a bug. That's a valid point! Maybe we create a new github label user_experience to capture issues like this vs runtime or implementation bugs? Might be a good way for users to gain merit through design contributions? I wonder if other projects have something like this? /cc @spotaws

It's going to be dropped along with the rest of the world in the bundle and we could read from that.

I think this is a good choice if we have a new plugin called "bundle manager" or something? This plugin could expose an API for getting all details about the bundle rather than modifying core OpenSearch with logic about the bundle (decouple from the core). I do have some questions about this, though:

  1. What is a bundle? A list of "blessed" plugins the project deems as first class?
  2. Bundle plugins are installed by default? But users can uninstall them if they don't want them?
  3. Does the "org" create a process for incubating third-party plugins with a well defined path to become a first class bundle plugin?
  4. Instead of a single blessed bundle, what about creating many bundles consisting of groups of similar plugins? (e.g., ML bundle, Spatial bundle, Observability bundle)? In this manner there's one opensearch w/ many experience options?
  5. Until we have a well defined version design, the many to one versioning is going to be an issue. How is it defined?

I have some more but I'll stick with these five for now...

stockholmux commented 3 years ago

I agree that a nice vision would be some sort of "choose your own adventure" OpenSearch, but that opens up a lot of considerations (dependency management/resolution, etc.).

From my personal POV, the bundled plugins are basically like a standard library in a programming language. Perhaps there are lessons in process & versioning from that world? I think that might inform some responses to @nknize's questions.

dblock commented 3 years ago

I propose we include the distribution version information in the OpenSearch.yml during distribution build and return that. This would allow someone to return any version from .yml but one can also do this in other ways today.

stockholmux commented 2 years ago

Is this fixed for the upcoming 1.1.1 release?

dagneyb commented 2 years ago

@dblock did this get fixed?

dblock commented 2 years ago

@dagneyb Nope. It's really a new feature where we want to report the version of the whole distribution separately. I think we need that information published from within the distribution mechanism.