AWS DependencyViolation errors when destroying sequential dependent resources

[wrightp]

This is more of a question on how we should best handle destroying dependent resources when the first resource hasn't reached the required state in order for the following resource to initiate a delete.

We may not always want to enforce waiting for resources to reach the expected state before moving on the next chef resource in a recipe.

For aws_ebs_volumes I added status waits https://github.com/chef/chef-provisioning-aws/blob/master/lib/chef/provider/aws_ebs_volume.rb#L43

Perhaps deleting a resource shouldn't wait unless an attribute is set.

  aws_ebs_volume volume do
    action :destroy
    wait_for_status true
  end

This is only an example of waiting for state, not resource dependency.

A solution for that would be to check for any dependent resources and add a :purge action to any resources where this applies and remove any dependent resources and wait for the required states. Or again, add a wait_for_status attribute to enable status waits.

Example recipe. See comments.

# All resources below are created successfully 
require 'chef/provisioning/aws_driver'

with_driver 'aws::us-west-2'

aws_key_pair 'ref-key-pair-eni'

aws_dhcp_options 'ref-dhcp-options-eni'

aws_vpc 'ref-vpc-eni' do
  cidr_block '10.0.0.0/24'
  internet_gateway true
  main_routes '0.0.0.0/0' => :internet_gateway
  dhcp_options 'ref-dhcp-options-eni'
end

aws_security_group 'ref-sg1-eni' do
  vpc 'ref-vpc-eni'
  inbound_rules '0.0.0.0/0' => 22
end

aws_security_group 'ref-sg2-eni' do
  vpc 'ref-vpc-eni'
  inbound_rules 'ref-sg1-eni' => 2224
  outbound_rules 2224 => 'ref-sg1-eni'
end

aws_route_table 'ref-public-eni' do
  vpc 'ref-vpc-eni'
  routes '0.0.0.0/0' => :internet_gateway
end

aws_subnet 'ref-subnet-eni' do
  vpc 'ref-vpc-eni'
  map_public_ip_on_launch true
  route_table 'ref-public-eni'
end

with_machine_options :bootstrap_options => { 
    :subnet_id => 'ref-subnet-eni',
    :key_name => 'ref-key-pair-eni',
    :security_group_ids => ['ref-sg1-eni', 'ref-sg2-eni']
  }

machine 'ref-machine-eni-1'

aws_network_interface 'ref-eni-1' do
  subnet 'ref-subnet-eni'
  security_groups ['ref-sg1-eni', 'ref-sg2-eni']
end

aws_network_interface 'ref-eni-1' do
  machine 'ref-machine-eni-1'
end

aws_network_interface 'ref-eni-1' do
  machine false
end

aws_network_interface 'ref-eni-1' do
  action :destroy
end

# termination sequence is initiated
machine 'ref-machine-eni-1' do
  action :destroy
end

# raises exception
#    AWS::EC2::Errors::DependencyViolation
#    -------------------------------------
#    The subnet 'subnet-163d9661' has dependencies and cannot be deleted.
#
# since the instance and possibly the network interfaces are still processing the delete/terminate actions
aws_subnet 'ref-subnet-eni' do
  action :destroy
end

aws_route_table 'ref-public-eni' do
  action :destroy
end

aws_security_group 'ref-sg2-eni' do
  action :destroy
end

aws_security_group 'ref-sg1-eni' do
  action :destroy
end

aws_vpc 'ref-vpc-eni' do
  action :destroy
end

aws_dhcp_options 'ref-dhcp-options-eni' do
  action :destroy
end

[wrightp]

@tyler-ball @jkeiser

[tyler-ball]

@patrick-wright I'm working on code right now in https://github.com/chef/chef-provisioning-aws/pull/152 to add a :purge action to all the AWS resources. This will call :purge on any dependent resources, call delete on self, and then wait until the delete on self is finished. The recursive nature of this should ensure that we successfully wait for dependent resources before deleting.

[tyler-ball]

Quick update on this - I merged #152 but there are still some resources that need to be updated to support the purge action. Working on this now.

[tyler-ball]

I'm going to close this for now since I got the aws_full example working idempotently. I updated a lot of resources there to be idempotent. We can open issues for any new cases we come across.