opensearch-scaling-manager

Akhil-Nair-ML commented 1 year ago

Proposal description

The opensearch-scaling-manager is tool offering auto-scaling facilities to the opensearch clusters. It will be done in the following way: Horizontally scaling up/down the cluster by adding or removing the nodes from the opensearch clusters. This will happen when the traffic increases and decrease. The opensearch-scaling-manager will analyze the data ingestion and scale up/down automatically.

The initial scope of the project is to develop a solution with scaling up/down by one node with core team and then extend the community support to develop the other feature of it. The plan is to start developing a solution for AWS platform and slowly extend it to other platforms as well.

Key pain points or opportunity details

Existing customers were facing a difficulty to gracefully scale up/down the opensearch cluster with respect to increase/decrease of the data ingestion and search queries. Hence it is a great opportunity for us to develop a solution which can help customers to solve their problems.

Moreover, we are making it as an open-source project, so core part of the development is done by core team and then we can extend this to open-source community.

Objective of the proposed project

Following are the deliverables for the proposed project:

Solution to horizontal scale up/down opensearch nodes in the cluster with a given configuration. To start with, we will support AWS platform.
The solution will be extended to support multiple node scaling and Rule based cluster sizing in the future releases and the support for multiple platforms.

Existing alternatives & why I think they are not enough (with links to those projects)

https://github.com/zalando-incubator/es-operator - This solution supports operator for running Elasticsearch in Kubernetes. The elastic search cluster will be managed by the es-operator which is not desirable.
https://github.com/upmc-enterprises/elasticsearch-operator - Though the solution supports operator for running Elasticsearch on AWS, the elastic search cluster will be managed by the es-operator which is not desirable.

Ballpark effort estimate (X engineers for Y Months)

4 Engineers for 3-4 months.

Expected number of users/organizations of the proposed project (Submit supporting material, if it exists).

aswinbh commented 1 year ago

When you say "Existing customers" pain points, how many customers you work with are having this problem? did you discuss the solution which you are proposing here? How interested are they to use this scaling manager once the basic version is released?

aswinbh commented 1 year ago

Can you brief the pain point of the customer with more details? what is the scale of the impact? what is at stake for them if the status-quo continues? If basic version of scaling manager is released, how much of the impact which our known customers would get addressed? Give couple of examples (without naming the customer if you don't wish to).

aswinbh commented 1 year ago

did you discuss these pain points in open-search mailing lists or discussion forums? are people not aware of any other existing alternatives? If discussion forums don't have other alternatives, can you propose your basic version plan there and gauge the interest in the open-search community ? once you discuss in open-search forums, update this GIT issue accordingly and create a folder under proposal-docs for your proposal and attach the communication details.

ganeshkumar-sundararajan-ML commented 1 year ago

Based on the details from SF team, atleast 2 customers are having issues with the scaling. We would ask SF team to reach to them with the proposed solution post our brainstorming and design approach.

ganeshkumar-sundararajan-ML commented 1 year ago

Can you brief the pain point of the customer with more details? what is the scale of the impact? what is at stake for them if the status-quo continues? If basic version of scaling manager is released, how much of the impact which our known customers would get addressed? Give couple of examples (without naming the customer if you don't wish to).

Based on the feedback from customer, the horizontal scaling is a manual process of correlating multiple events and triggers and takes hours together. We are expecting that the solution will ease out the process of correlating the events and triggers and decide on the scaling. Still the scaling time might be same as it depends on the cluster size, number of shards and volume of data to rebalance.

ganeshkumar-sundararajan-ML commented 1 year ago

did you discuss these pain points in open-search mailing lists or discussion forums? are people not aware of any other existing alternatives? If discussion forums don't have other alternatives, can you propose your basic version plan there and gauge the interest in the open-search community ? once you discuss in open-search forums, update this GIT issue accordingly and create a folder under proposal-docs for your proposal and attach the communication details.

This is still in progress. Will try to have this posted once we have the design approach agreed upon.

maplelabs / FOSS-Projects