search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.88k stars 1.84k forks source link

Need Support for multi node cluster setup on local. #3933

Open pranikum opened 2 years ago

pranikum commented 2 years ago

Is your feature request related to a problem? Please describe. Currently we only have option to run a single node cluster on local for Opensearch. While testing we need to have provision to create a multi node cluster on local with zonal configuration. This will assist in testing features which requires multi node setup.

Background With 2859 we intend to weigh away shard search traffic, however since OpenSearch follows a synchronous replication model, it is possible to have replication request stuck due to any impairment in the write path. The current health check mechanisms to detect and remediate a bad node is only a best effort strategy and doesn't cover deeper health checks across all network paths. For predictability, we propose pulling an andon cord to cut-off inter-zone replication traffic, which can be achieved by decommissioning the node in the impacted zone.

For testing weigh away feature it will be really helpful to have the ability to create multi node cluster with multi zone support on local environment.

Describe the solution you'd like As a developer i would prefer to be able to start multi node cluster on local with zonal configuration. For example: ./gradlew run -PnodeCount=3 -PzoneCount=2

The framework should start multiple Opensearch nodes with zone configuration.

Describe alternatives you've considered Another approach could be to just add a project property to mention multi node configuration instead of specific configuration for node count or zone count and have a fixed configuration for multi node setup.

mgodwan commented 2 years ago

There is a docker distribution target which can be built locally to test such scenarios using docker compose.

:distribution:docker:composeUp

https://github.com/opensearch-project/OpenSearch/blob/main/distribution/docker/docker-compose.yml

navneet1v commented 2 years ago

When building the any plugin these arguments are supported not sure why they are not supported with the main OpenSearch Repo.

pranikum commented 2 years ago

There is a docker distribution target which can be built locally to test such scenarios using docker compose.

:distribution:docker:composeUp

https://github.com/opensearch-project/OpenSearch/blob/main/distribution/docker/docker-compose.yml

I have not tried the docker compose yet though. However gradle command still has benefits. We can configure the size of cluster and zones just based on command line properties.

For docker compose we need to update the file every time based on the cluster size. Also we can use debugging capabilities easily. Not sure on docker approach though for that. I hope that can be done with some more configuration.