elastic / azure-marketplace

Elasticsearch Azure Marketplace offering + ARM template
https://www.elastic.co/guide/en/elastic-stack-deploy/current/index.html
MIT License
120 stars 162 forks source link

Add support for Availability Zones to increase high availability #370

Open kbandrew opened 3 years ago

kbandrew commented 3 years ago

The deployed VMs are grouped into Availability Sets, but for increased availability, it would be good if there could be support for distributing VMs across Availability Zones.

I believe this is on the roadmap: -

The template does not currently use Availability zones since they are not yet available in all Azure regions to which the template can be deployed. It is expected that the template will use Availability zones in the future.

Availability Zones are now well supported across Azure regions. Perhaps the support could be available via a parameter in the template if support across ALL regions is a concern?

NOTE: Moving VMs to Availability Zones isn't a trivial task, so we are now having to consider using a different approach to creating our elastic clusters.

Mpdreamz commented 3 years ago

@kbandrew the original reservations to do this OOTB by default still stands:

The situations has not changed since

https://docs.microsoft.com/en-us/azure/availability-zones/az-region

We would definitely be open to a PR that adds support to it through a flag but realistically we won't add support for this any time soon I'm afraid.

kbandrew commented 3 years ago

@Mpdreamz I wouldn't mind creating a PR if you are open to a contribution on this.

Is it relatively easy to run the main template against a local copy of the files vs running against https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/ ?

Mpdreamz commented 3 years ago

Awesome!

It is, your copy has to be up on github (fork) though the docs here go into how to set it up:

https://github.com/elastic/azure-marketplace/tree/master/build

You can add a test case here:

https://github.com/elastic/azure-marketplace/tree/master/build/arm-tests

And run the integration tests on your deploy with:

npm run deploy -- --test <your test case>

That will deploy the test in a new resource group validate everything starts and then teardown.

(On phone but more then happy to expand on this if needed :))

kbandrew commented 3 years ago

Thanks @Mpdreamz, that is very helpful.

We have added in support for Availability Zones and have successfully created a new cluster to our spec! We will look at making it fully parameterised, polish it up and make sure the test cases pass before we submit a PR.