search

chef-boneyard / chef-provisioning-aws

AWS driver and resources for Chef that uses the AWS SDK
Apache License 2.0
142 stars 122 forks source link

AWS VPC Provisioning Fails. Chef returns error that it is not able to find resource with resource ID #512

Open r4hulgupta opened 7 years ago

r4hulgupta commented 7 years ago

Recently updated Chef-Client to 12.15.19 and started seeing this new error. I have been using my VPC creation recipes for long and now when I run it, Chef tries to check on AWS for a resource with it's ID but returns not found even when the resource exists on AWS. Please help! Chef Stacktrace Output:

Generated at 2016-11-08 23:14:40 +0000
AWS::EC2::Errors::InvalidNetworkAclID::NotFound: aws_network_acl[EG5988_SEC_21.55_ACL_01] (no AWS object id) (network_stack::network_stack line 170) had an error: AWS::EC2::Errors::InvalidNetworkAclID::NotFound: The networkAcl ID 'acl-d837ecbf' does not exist
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `create_tags'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/tag_collection.rb:64:in `create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/resource_tag_collection.rb:88:in `[]='
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:28:in `block (2 levels) in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:293:in `block in retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/retryable-2.0.4/lib/retryable.rb:68:in `retryable'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:291:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:298:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:27:in `block in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:42:in `block in converge_by'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/mixin/why_run.rb:52:in `add_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:176:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:41:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:19:in `create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:123:in `block in <class:AWSProvider>'
(eval):2:in `block in action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `instance_eval'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `compile_and_converge_action'
(eval):2:in `action_create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:11:in `action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:145:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource.rb:603:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:69:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block (2 levels) in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `each'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:96:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:669:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `catch'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:703:in `converge_and_save'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:283:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:284:in `run_with_graceful_exit_option'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:260:in `block in run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/local_mode.rb:44:in `with_server_connectivity'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:243:in `run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `loop'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:434:in `run_application'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:60:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chefdk/bin/chef-client:68:in `load'
C:/opscode/chefdk/bin/chef-client:68:in `<main>'

>>>> Caused by AWS::EC2::Errors::InvalidNetworkAclID::NotFound: The networkAcl ID 'acl-d837ecbf' does not exist
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `create_tags'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/tag_collection.rb:64:in `create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/aws-sdk-v1-1.66.0/lib/aws/ec2/resource_tag_collection.rb:88:in `[]='
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:28:in `block (2 levels) in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:293:in `block in retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/retryable-2.0.4/lib/retryable.rb:68:in `retryable'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:291:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:298:in `retry_with_backoff'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:27:in `block in create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:42:in `block in converge_by'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/mixin/why_run.rb:52:in `add_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:176:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:41:in `converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:19:in `create_aws_object'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provisioning/aws_driver/aws_provider.rb:123:in `block in <class:AWSProvider>'
(eval):2:in `block in action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `instance_eval'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:361:in `compile_and_converge_action'
(eval):2:in `action_create'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-aws-2.0.0/lib/chef/provider/aws_network_acl.rb:11:in `action_create'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/provider.rb:145:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource.rb:603:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:69:in `run_action'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block (2 levels) in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `each'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:97:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/runner.rb:96:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:669:in `block in converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `catch'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:664:in `converge'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:703:in `converge_and_save'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/client.rb:283:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:284:in `run_with_graceful_exit_option'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:260:in `block in run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/local_mode.rb:44:in `with_server_connectivity'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:243:in `run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `loop'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:450:in `interval_run_chef_client'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application/client.rb:434:in `run_application'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/lib/chef/application.rb:60:in `run'
C:/Users/rgupta/AppData/Local/chefdk/gem/ruby/2.3.0/gems/chef-12.15.19-universal-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chefdk/bin/chef-client:68:in `load'
C:/opscode/chefdk/bin/chef-client:68:in `<main>'
r4hulgupta commented 7 years ago

Works again after downgrading Chef client to 12.14.89.

UPDATE: Worked one time then again error. Gives errors for different things, like ACL not found with ID, Route Table not found with ID...

r4hulgupta commented 7 years ago

So this issue doesn't seem to be due to any versions. It is intermittent and I can still see this. Sometimes Chef is not able to find the resource even if it is created, mostly seen when it tries to go and create Name tags to the resource. Any solution to this? Is there a way to put a little delay in retries?