AWS::Elasticsearch::Domain - Auto-Tune

[K-Stefan]

Scope of request

ES got a new feature Auto-tune. This feature with all its options are not supported by AWS Cloudformation.

Options missing:

• Auto-Tune: Enabled State
• Maintenance Windows: State
• ScheduledAutoTuneDetails

API Specification - CreateElasticSearchDomain - AutoTuneOptions

{'AutoTuneOptions': {'DesiredState': 'ENABLED | DISABLED',
                     'MaintenanceSchedules': [{'CronExpressionForRecurrence': 'string',
                                               'Duration': {'Unit': 'HOURS',
                                                            'Value': 'long'},
                                               'StartAt': 'timestamp'}]}}

Expected behavior

Configure the options for ES Auto-tune via Cloudformation - see Screenshot of console:

Suggest specific test cases

none

Helpful Links to speed up research and evaluation

• New Feature Announcement: https://aws.amazon.com/about-aws/whats-new/2021/03/amazon-elasticsearch-service-announces-auto-tune-feature-improved-performance-application-availability/
• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticsearch-domain.html
• https://awsapichanges.info/archive/changes/66ea51-es.html

Category (required)

• Analytics (Athena, EMR, Glue,...)

[ChristopheBougere]

Ability to set Auto-tune in CloudFormation is, indeed, required to make it available in CDK as well. See https://github.com/aws/aws-cdk/issues/16575

[xaoyuli]

Confirmed with the provider team. This is a new feature, not a bug fix. So change it to enhancement.

[ansjcy]

We have fixed this issue, could you please verify if we can close this ticket? Thank you!

[SamStephens]

@ansjcy did you comment on the wrong issue? I don't see any fix/change here.

[SamStephens]

Note that the inability to set Auto-Tune prevents usage of T3 instances, because they're incompatible with Auto-Tune.

[StuartBertram]

When did this change? We deployed a T3 instance about four months ago and it was fine. I made one change to the config, it tried to redeploy, and now I'm getting the "not supported" error with no obvious way of disabling it!

It's only a dev instance for testing, so I really don't care about autotune. But if it isn't possible to do it in CloudFormation then my 7c/h instance suddenly becomes a 50% more expensive 11c/h c6g.large instance.

[huevos-y-bacon]

When did this change? We deployed a T3 instance about four months ago and it was fine. I made one change to the config, it tried to redeploy, and now I'm getting the "not supported" error with no obvious way of disabling it!

It's only a dev instance for testing, so I really don't care about autotune. But if it isn't possible to do it in CloudFormation then my 7c/h instance suddenly becomes a 50% more expensive 11c/h c6g.large instance.

Did you get to the bottom of this in the end? Any sort of answer? I've just run into the same issue with a domain that's been running quite happily for a long time, now needing a minor update and rolls back because of this.

[StuartBertram]

The answer we got was "it's not supported on those instances and it shouldn't have been enabled". You can still use the smaller instance types, you just can't use autotune with them. And if the instance predates autotune and incorrectly got it enabled when it was released then you can't update through CloudFormation until you manually fix it through the web console.

[v4de]

Trying to scale down an opensearch cluster with cloudformation and we are getting the same error. No way to turn off auto-tune with cloudformation currently. We also never enabled this feature when we scaled up for performance testing.

Resource handler returned message: "Invalid request provided: Autotune is not supported in t2/t3 instance types. Disable autotune or change your instance type. (Service: OpenSearch, Status Code: 400, Request ID: xxxx-xxxx-xxxx-xxxx-xxxx)" (RequestToken: xxxx-xxxx-xxxx-xxxx-xxxx, HandlerErrorCode: InvalidRequest)

[tibbe]

As I elaborated a bit more on in re:Post, 70% of our AWS development costs are due to OpenSearch CF only allowing us to run on large instances. It would be really nice to be able to turn off auto-tune and thus run on small instances.

[StuartBertram]

@tibbe - I'm running one of my dev instances on a pair of t3.medium.search. It is possible. You just need to either a) have a new instance or b) manually disable auto-tune through the console and then make the change in CloudFormation/Serverless.

[tibbe]

@tibbe - I'm running one of my dev instances on a pair of t3.medium.search. It is possible. You just need to either a) have a new instance or b) manually disable auto-tune through the console and then make the change in CloudFormation/Serverless.

I really don't want to do (b). It defeats the point of having infrastructure as code. (b) is interesting. It goes against what the AWS docs say.

[omriman12]

You can always use a CR to do it if you want to keep it fully CDK/Cloudformation managed. We choose to run a 1 time script in all our accounts, as this is a 1 time thing.