Cannot Switch Back to Instance Storage From EBS

[etherops]

Expected Behavior:

It is possible to switch back and for between ephemeral/ebs volume storage using this module

Actual Behavior:

It is not possible to switch back to ephemeral volume storage once cluster is configured for EBS storage.

Details

After modifying the cluster such that is it EBS backed it is impossible to switch back to Instance storage using this terraform module.

• ~Note: I did make the switch to EBS in the console, and not in terraform. I am not sure of the behevior had I made the switch in terraform code. But please bear with me, there is still a simple fix for this.~ <-- True but irrelevant, see 1st comment below

The problem is that after switching back to instance storage, AWS only switchd the EBSEnabled parameter back on the cluster config, but not the VolumeSize nor VolumeType.

For example:

aws es update-elasticsearch-domain-config --domain-name tf-production-pelias-es-c4 --ebs-options EBSEnabled=false
....

        "EBSOptions": {
            "Status": {
                "PendingDeletion": false, 
                "State": "Processing", 
                "CreationDate": 1529706898.069, 
                "UpdateVersion": 34, 
                "UpdateDate": 1530371848.418
            }, 
            "Options": {
                "VolumeSize": 96, 
                "VolumeType": "gp2", 
                "EBSEnabled": false
            }
        }, 

This causes a conflict with logic in the terraform module, then. When you remove the VolumeSize, config, for example, terraform then attempts to change the volume size from 96 -> 0, as this:

for:

  ebs_volume_type       = ""
#  ebs_volume_size       = "96"

terraform apply:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.pelias_es_c4.aws_elasticsearch_domain.es_vpc
      ebs_options.0.volume_size: "96" => "0"

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.pelias_es_c4.aws_elasticsearch_domain.es_vpc: Modifying... (ID: arn:aws:es:us-east-1:666665886686:domain/tf-production-pelias-es-c4)
  ebs_options.0.volume_size: "96" => "0"

Error: Error applying plan:

1 error(s) occurred:

* module.pelias_es_c4.aws_elasticsearch_domain.es_vpc: 1 error(s) occurred:

* aws_elasticsearch_domain.es_vpc: ValidationException: EBSEnabled must be set to true to use any of the EBS options.
    status code: 400, request id: 495f0ec8-7c7e-11e8-9630-31af94696b3e

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

however, if you then attempt to set volume size to 96 to perhaps avoid this problem:

  ebs_volume_type       = ""
  ebs_volume_size       = "96"

terraform apply:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.pelias_es_c4.aws_elasticsearch_domain.es_vpc
      ebs_options.0.ebs_enabled: "false" => "true"

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

the module then attempts to set EBSEnabled back to true, which we do not want!

The reason i've discovered is this line of logic in the main_vpc file.

    ebs_enabled = "${var.ebs_volume_size > 0 ? true : false}"

I understand why this was done originally... however because of the behavior of leaving the settings lingering after EBSEnabled is set to false, it is necessary to allow users to pass in ebs_enabled instead of computing it from ebs_volume_size to avoid this catch 22 scenario.

[etherops]

Additionally,

reconfiguring the cluster as EBSEnabled via terraform, and attempting to switch back by removing all EBS Options from the module inputs results in the following same situation, so it does not look as if this is a result of drift happening by modifying the cluster in the console.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.pelias_es_c4.aws_elasticsearch_domain.es_vpc
      ebs_options.0.ebs_enabled: "true" => "false"
      ebs_options.0.volume_size: "96" => "0"

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.pelias_es_c4.aws_elasticsearch_domain.es_vpc: Modifying... (ID: arn:aws:es:us-east-1:666665886686:domain/tf-production-pelias-es-c4)
  ebs_options.0.ebs_enabled: "true" => "false"
  ebs_options.0.volume_size: "96" => "0"

Error: Error applying plan:

1 error(s) occurred:

* module.pelias_es_c4.aws_elasticsearch_domain.es_vpc: 1 error(s) occurred:

* aws_elasticsearch_domain.es_vpc: ValidationException: EBSEnabled must be set to true to use any of the EBS options.
    status code: 400, request id: c798ca14-7c7f-11e8-9fce-ff1b86a665e0

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

[etherops]

As a follow up, I took an initial stab at a PR to fix only to find out that under the hood AWS does not remove the volume_size setting even when you switch away from EBS. This causes too much complexity to programmatically account for on the terraform side of things, and so I am closing.