search

sous-chefs / aws

Development repository for the aws cookbook
https://supermarket.chef.io/cookbooks/aws
Apache License 2.0
606 stars 553 forks source link
aws chef chef-cookbook chef-resource hacktoberfest managed-by-terraform

readme

aws Cookbook

Cookbook Version CI State OpenCollective OpenCollective License

Overview

This cookbook provides resources for configuring and managing nodes running in Amazon Web Services as well as several AWS service offerings.

Included resources:

Unsupported AWS resources that have other cookbooks include but are not limited to:

Important - Security Implications

Please review any and all security implications of using any of these resources. This cookbook presents resources which could easily be poorly implemented, abused or exploited.

You will want to understand any and all security implications and architect your implementation accordingly before proceeding.

Some recommendations are below:

See iam_restrictions_and_conditions

Maintainers

This cookbook is maintained by the Sous Chefs. The Sous Chefs are a community of Chef cookbook maintainers working together to maintain important cookbooks. If you’d like to know more please visit sous-chefs.org or come chat with us on the Chef Community Slack in #sous-chefs.

Requirements

Platforms

Chef

Cookbooks

Credentials

In order to manage AWS components, authentication credentials need to be available to the node. There are 3 ways to handle this:

  1. Explicitly set the credentials when using the resources
  2. Use the credentials in the ~/.aws/credentials file
  3. Let the resource pick up credentials from the IAM role assigned to the instance

Also new resources can now assume an STS role, with support for MFA as well. Instructions are below in the relevant section.

Using resource parameters

In order to pass the credentials to the resource, credentials must be available to the node. There are a number of ways to handle this, such as node attributes applied to the node or via Chef roles/environments.

We recommend storing these in an encrypted databag, and loading them in the recipe where the resources are used.

Example Data Bag:

% knife data bag show aws main
{
  "id": "main",
  "aws_access_key_id": "YOUR_ACCESS_KEY",
  "aws_secret_access_key": "YOUR_SECRET_ACCESS_KEY",
  "aws_session_token": "YOUR_SESSION_TOKEN"
}

This can be loaded in a recipe with:

aws = data_bag_item('aws', 'main')

And to access the values:

aws['aws_access_key_id']
aws['aws_secret_access_key']
aws['aws_session_token']

We'll look at specific usage below.

Using local credentials

If credentials are not supplied via parameters, resources will look for the credentials in the ~/.aws/credentials file:

[default]
aws_access_key_id = ACCESS_KEY_ID
aws_secret_access_key = ACCESS_KEY

Note that this also accepts other profiles if they are supplied via the ENV['AWS_PROFILE'] environment variable.

Using IAM instance role

If your instance has an IAM role, then the credentials can be automatically resolved by the cookbook using Amazon instance metadata API.

You can then omit the authentication properties aws_secret_access_key and aws_access_key when using the resource.

Of course, the instance role must have the required policies. Here is a sample policy for EBS volume management:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:AttachVolume",
        "ec2:CreateVolume",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyVolumeAttribute",
        "ec2:DescribeVolumeAttribute",
        "ec2:DescribeVolumeStatus",
        "ec2:DescribeVolumes",
        "ec2:DetachVolume",
        "ec2:EnableVolumeIO"
      ],
      "Sid": "Stmt1381536011000",
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    }
  ]
}

For resource tags:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:CreateTags",
        "ec2:DescribeTags"
      ],
      "Sid": "Stmt1381536708000",
      "Resource": [
        "*"
      ],
      "Effect": "Allow"
    }
  ]
}

Assuming roles via STS and using MFA

The following is an example of how roles can be assumed using MFA. The following can also be used to assumes roles that do not require MFA, just ensure that the MFA arguments (serial_number and token_code) are omitted.

This assumes you have also stored the cfn_role_arn, and mfa_serial attributes as well, but there are plenty of ways these attributes can be supplied (they could be stored locally in the consuming cookbook, for example).

Note that MFA codes cannot be recycled, hence the importance of creating a single STS session and passing that to resources. If multiple roles need to be assumed using MFA, it is probably prudent that these be broken up into different recipes and chef-client runs.

require 'aws-sdk-core'
require 'securerandom'

session_id = SecureRandom.hex(8)
sts = ::Aws::AssumeRoleCredentials.new(
  client: ::Aws::STS::Client.new(
    credentials: ::Aws::Credentials.new(
      node['aws']['aws_access_key_id'],
      node['aws']['aws_secret_access_key']
    ),
    region: 'us-east-1'
  ),
  role_arn: node['aws']['cfn_role_arn'],
  role_session_name: session_id,
  serial_number: node['aws']['mfa_serial'],
  token_code: node['aws']['mfa_code']
)

aws_cloudformation_stack 'kitchen-test-stack' do
  action :create
  template_source 'kitchen-test-stack.tpl'
  aws_access_key sts.access_key_id
  aws_secret_access_key sts.secret_access_key
  aws_session_token sts.session_token
end

When running the cookbook, ensure that an attribute JSON is passed that supplies the MFA code. Example using chef-zero:

echo '{ "aws": { "mfa_code": "123456" } }' > mfa.json && chef-client -z -o 'recipe[aws_test]' -j mfa.json

Running outside of an AWS instance

region can be specified on each resource if the cookbook is being run outside of an AWS instance. This can prevent some kinds of failures that happen when resources try to detect region.

aws_cloudformation_stack 'kitchen-test-stack' do
  action :create
  template_source 'kitchen-test-stack.tpl'
  region 'us-east-1'
end

Resources

aws_cloudformation_stack

Manage CloudFormation stacks.

Actions

Properties

Examples

aws_cloudformation_stack 'example-stack' do
  region 'us-east-1'
  template_source 'example-stack.tpl'
  parameters ([
    {
      :parameter_key => 'KeyPair',
      :parameter_value => 'user@host'
    },
    {
      :parameter_key => 'SSHAllowIPAddress',
      :parameter_value => '127.0.0.1/32'
    }
  ])
end

aws_cloudwatch

Use this resource to manage CloudWatch alarms.

Actions

Properties

For more information about parameters, see CloudWatch Identifiers in the Using CloudWatch guide.

Examples

aws_cloudwatch "kitchen_test_alarm" do
  period 21600
  evaluation_periods 2
  threshold 50.0
  comparison_operator "LessThanThreshold"
  metric_name "CPUUtilization"
  namespace "AWS/EC2"
  statistic "Maximum"
  dimensions [{"name" : "InstanceId", "value" : "i-xxxxxxx"}]
  action :create
end

aws_dynamodb_table

Use this resource to create and delete DynamoDB tables. This includes the ability to add global secondary indexes to existing tables.

Actions

Properties

Several of the attributes shown here take parameters as shown in the AWS Ruby SDK Documentation. Also, the AWS DynamoDB Documentation may be of further help as well.

Examples

aws_dynamodb_table 'example-table' do
  action :create
  attribute_definitions [
    { attribute_name: 'Id', attribute_type: 'N' },
    { attribute_name: 'Foo', attribute_type: 'N' },
    { attribute_name: 'Bar', attribute_type: 'N' },
    { attribute_name: 'Baz', attribute_type: 'S' }
  ]
  key_schema [
    { attribute_name: 'Id', key_type: 'HASH' },
    { attribute_name: 'Foo', key_type: 'RANGE' }
  ]
  local_secondary_indexes [
    {
      index_name: 'BarIndex',
      key_schema: [
        {
          attribute_name: 'Id',
          key_type: 'HASH'
        },
        {
          attribute_name: 'Bar',
          key_type: 'RANGE'
        }
      ],
      projection: {
        projection_type: 'ALL'
      }
    }
  ]
  global_secondary_indexes [
    {
      index_name: 'BazIndex',
      key_schema: [{
        attribute_name: 'Baz',
        key_type: 'HASH'
      }],
      projection: {
        projection_type: 'ALL'
      },
      provisioned_throughput: {
        read_capacity_units: 1,
        write_capacity_units: 1
      }
    }
  ]
  provisioned_throughput ({
    read_capacity_units: 1,
    write_capacity_units: 1
  })
  stream_specification ({
    stream_enabled: true,
    stream_view_type: 'KEYS_ONLY'
  })
end

aws_ebs_volume

The resource only handles manipulating the EBS volume, additional resources need to be created in the recipe to manage the attached volume as a filesystem or logical volume.

Actions

Properties

Examples

Create a 50G volume, attach it to the instance as /dev/sdi:

aws_ebs_volume 'db_ebs_volume' do
  size 50
  device '/dev/sdi'
  action [:create, :attach]
end

Create a new 50G volume from the snapshot ID provided and attach it as /dev/sdi.

aws_ebs_volume 'db_ebs_volume_from_snapshot' do
  size 50
  device '/dev/sdi'
  snapshot_id 'snap-ABCDEFGH'
  action [:create, :attach]
end

aws_elastic_ip

The elastic_ip resource provider does not support allocating new IPs. This must be done before running a recipe that uses the resource. After allocating a new Elastic IP, we recommend storing it in a databag and loading the item in the recipe.

Actions

Properties

Examples

aws_elastic_ip '34.15.30.10' do
  action :associate
end

aws_elastic_ip 'Server public IP' do
  ip '34.15.30.11'
  action :associate
end

aws_elastic_lb

elastic_lb handles registering and removing nodes from ELBs. The resource also adds basic support for creating and deleting ELBs. Note that currently this resource is not fully idempotent so it will not update the existing configuration of an ELB.

Actions

Properties

Examples

ELB running in classic networking listening on port 80.

aws_elastic_lb 'Setup the ELB' do
  name 'example-elb'
  action :create
  availability_zones ['us-west-2a']
  listeners [
    {
      instance_port: 80,
      instance_protocol: 'HTTP',
      load_balancer_port: 80,
      protocol: 'HTTP',
    },
  ]
end

To register the node in the 'QA' ELB:

aws_elastic_lb 'elb_qa' do
  name 'QA'
  action :register
end

aws_iam_user

Use this resource to manage IAM users.

Actions

Properties

The IAM user takes the name of the resource. A path can be specified as well. For more information about paths, see IAM Identifiers in the Using IAM guide.

Examples

aws_iam_user 'example-user' do
  action :create
  path '/'
end

aws_iam_group

Use this resource to manage IAM groups. The group takes the name of the resource.

Actions

Properties

Examples

aws_iam_group 'example-group' do
  action :create
  path '/'
  members [
    'example-user'
  ]
  remove_members true
  policy_members [
    'arn:aws:iam::123456789012:policy/example-policy'
  ]
  remove_policy_members true
end

aws_iam_policy

Use this resource to create an IAM policy. The policy takes the name of the resource.

Actions

Properties

Examples

aws_iam_policy 'example-policy' do
  action :create
  path '/'
  account_id '123456789012'
  policy_document <<-EOH.gsub(/^ {4}/, '')
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1234567890",
                "Effect": "Allow",
                "Action": [
                    "sts:AssumeRole"
                ],
                "Resource": [
                    "arn:aws:iam::123456789012:role/example-role"
                ]
            }
        ]
    }
  EOH
end

aws_iam_role

Use this resource to create an IAM role. The policy takes the name of the resource.

Actions

Properties

Examples

aws_iam_role 'example-role' do
  action :create
  path '/'
  policy_members [
    'arn:aws:iam::123456789012:policy/example-policy'
  ]
  remove_policy_members true
  assume_role_policy_document <<-EOH.gsub(/^ {4}/, '')
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "",
          "Effect": "Deny",
          "Principal": {
            "AWS": "*"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  EOH
end

aws_instance_monitoring

Allows detailed CloudWatch monitoring to be enabled for the current instance.

Actions

Properties

Examples

aws_instance_monitoring "enable detailed monitoring"

aws_instance_role

Used to associate an IAM role (by way of an IAM instance profile) with an instance. Replaces the instance's current role association if one already exists.

Actions

Properties

Requirements

IAM permisions:

Examples

aws_instance_role "change to example role" do
  profile_arn 'arn:aws:iam::123456789012:instance-profile/ExampleInstanceProfile'
end

aws_instance_term_protection

Allows termination protection (AKA DisableApiTermination) to be enabled for an instance.

Actions

Properties

Examples

aws_instance_term_protection "enable termination protection"

aws_kinesis_stream

Use this resource to create and delete Kinesis streams. Note that this resource cannot be used to modify the shard count as shard splitting is a somewhat complex operation (for example, even CloudFormation replaces streams upon update).

Actions

Properties

Examples

aws_kinesis_stream 'example-stream' do
 action :create
 starting_shard_count 1
end

aws_resource_tag

resource_tag can be used to manipulate the tags assigned to one or more AWS resources, i.e. ec2 instances, EBS volumes or EBS volume snapshots.

Actions

Properties

Examples

Assigning tags to a node to reflect its role and environment:

aws_resource_tag node['ec2']['instance_id'] do
  tags('Name' => 'www.example.com app server',
       'Environment' => node.chef_environment)
  action :update
end

Assigning a set of tags to multiple resources, e.g. ebs volumes in a disk set:

aws_resource_tag 'my awesome raid set' do
  resource_id ['vol-d0518cb2', 'vol-fad31a9a', 'vol-fb106a9f', 'vol-74ed3b14']
  tags('Name' => 'My awesome RAID disk set',
       'Environment' => node.chef_environment)
end
aws_resource_tag 'db_ebs_volume' do
  resource_id lazy { node['aws']['ebs_volume']['db_ebs_volume']['volume_id'] }
  tags ({ 'Service' => 'Frontend' })
end

aws_route53_record

Actions

Properties

Examples

Create a simple record

route53_record "create a record" do
  name  "test"
  value "16.8.4.2"
  type  "A"
  weight "1"
  set_identifier "my-instance-id"
  zone_id "ID VALUE"
  overwrite true
  fail_on_error false
  action :create
end

Delete an existing record. Note that value is still necessary even though we're deleting. This is a limitation in the AWS SDK.

aws_route53_record "delete a record" do
  name  "test"
  value "16.8.4.2"
  type 'A'
  value '123'
  action :delete
end

aws_route53_zone

Actions

Properties

Examples

aws_route53_zone 'testkitchen.dmz' do
  description 'My super important zone'
  action :create
end

aws_secondary_ip.rb

This feature is available only to instances within VPCs. It allows you to assign multiple private IP addresses to a network interface.

Actions

Properties

aws_s3_file

s3_file can be used to download a file from s3 that requires aws authorization. This is a wrapper around the core chef remote_file resource and supports the same resource attributes as remote_file. See [remote_file Chef Docs] (https://docs.chef.io/resource_remote_file.html) for a complete list of available attributes.

Properties

Actions

Examples

aws_s3_file '/tmp/foo' do
  bucket 'i_haz_an_s3_buckit'
  remote_path 'path/in/s3/bukket/to/foo'
  region 'us-west-1'
end
aws_s3_file '/tmp/bar' do
  bucket 'i_haz_another_s3_buckit'
  remote_path 'path/in/s3/buckit/to/foo'
  region 'us-east-1'
  requester_pays true
end

aws_s3_bucket

s3_bucket can be used to create or delete S3 buckets. Note that buckets can only be deleted if they are empty unless you specify delete_all_objects true, which will delete EVERYTHING in your bucket first.

Actions

Properties

Examples

aws_s3_bucket 'some-unique-name' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  versioning true
  region 'us-west-1'
  action :create
end
aws_s3_bucket 'another-unique-name' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  region 'us-west-1'
  action :delete
end

aws_secondary_ip

The secondary_ip resource provider allows one to assign/un-assign multiple private secondary IPs on an instance within a VPC. The number of secondary IP addresses that you can assign to an instance varies by instance type. If no ip address is provided on assign, a random one from within the subnet will be assigned. If no interface is provided, the default interface as determined by Ohai will be used.

Examples

aws_secondary_ip 'assign_additional_ip' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  ip ip_info['private_ip']
  interface 'eth0'
  action :assign
end

aws_security_group

security_group can be used to create or update security groups and associated rules.

Actions

Properties

Tags

Ingress/Egress rules

Note - this manages ALL rules on the security group. Any exist rules not included in these definitions will be removed.

Examples

aws_security_group 'some-unique-name' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  description 'some-unique-description'
  vpc_id 'vpc-000000000'
  ip_permissions []
  ip_permissions_egress []
  tags []
  action :create
end

Manages tags

aws_security_group 'some-unique-name' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  description 'some-unique-description'
  vpc_id 'vpc-000000000'
  ip_permissions []
  ip_permissions_egress []
  tags [{ key: 'tag_key', value: 'tag_value' }]
  action :create
end

Manages ingress/egress rules

aws_security_group 'some-unique-name' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  description 'some-unique-description'
  vpc_id 'vpc-000000000'
  ip_permissions [{
                   from_port: 22,
                   ip_protocol: 'tcp',
                   ip_ranges: [
                     {
                       cidr_ip: '10.10.10.10/24',
                       description: 'SSH access from the office',
                     },
                   ],
                   to_port: 22,
                  }]
  ip_permissions_egress [{
                   from_port: 123,
                   ip_protocol: 'udp',
                   ip_ranges: [
                     {
                       cidr_ip: '10.10.10.10/24',
                       description: 'ntp from the office',
                     },
                   ],
                   to_port: 123,
                        }]
  action :create
end

Alternatively you can use the class definitions for a more strongly typed object

aws_security_group 'some-unique-name' do
  aws_access_key aws['aws_access_key_id']
  aws_secret_access_key aws['aws_secret_access_key']
  description 'some-unique-description'
  vpc_id 'vpc-000000000'
  ip_permissions [Aws::EC2::Types::IpPermission.new.to_h]
  ip_permissions_egress [Aws::EC2::Types::IpPermission.new.to_h]
  action :create
end

aws_ssm_parameter_store

The ssm_parameter_store resource provider allows one to get, create and delete keys and values in the AWS Systems Manager Parameter Store. Values can be stored as plain text or as an encrypted string. In order to use the paramater store resource your ec2 instance role must have the proper policy. This sample policy allows get, creating and deleting parameters. You can adjust the policy to your needs. It is recommended that you have one role with the ability to create secrets and another that can only read the secrets. It is important to set sensitive true in the resources where the secrets are used so that secrets are not exposed in log files.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:RemoveTagsFromResource",
                "ssm:GetParameterHistory",
                "ssm:AddTagsToResource",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:DeleteParameters"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:document/*",
                "arn:aws:ssm:*:*:parameter/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "ssm:DescribeParameters",
            "Resource": "*"
        }
    ]
}

Actions

Properties

Examples

Create String Parameter
aws_ssm_parameter_store 'create testkitchen record' do
  path 'testkitchen'
  description 'testkitchen'
  value 'testkitchen'
  type 'String'
  action :create
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end
Create Encrypted String Parameter with Custom KMS Key
aws_ssm_parameter_store "create encrypted test kitchen record" do
  path '/testkitchen/EncryptedStringCustomKey'
  description 'Test Kitchen Encrypted Parameter - Custom'
  value 'Encrypted Test Kitchen Custom'
  type 'SecureString'
  key_id '5d888999-5fca-3c71-9929-014a529236e1'
  action :create
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
 end
Delete Parameter
aws_ssm_parameter_store 'delete testkitchen record' do
  path 'testkitchen'
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
  action :delete
end
Get Parameters and Populate Template
aws_ssm_parameter_store 'get clear_value' do
  path '/testkitchen/ClearTextString'
  return_key 'clear_value'
  action :get
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end

aws_ssm_parameter_store 'get decrypted_value' do
  path '/testkitchen/EncryptedStringDefaultKey'
  return_key 'decrypted_value'
  with_decryption true
  action :get
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end

aws_ssm_parameter_store 'get decrypted_custom_value' do
  path '/testkitchen/EncryptedStringCustomKey'
  return_key 'decrypted_custom_value'
  with_decryption true
  action :get
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end

aws_ssm_parameter_store 'getParameters' do
  path ['/testkitchen/ClearTextString', '/testkitchen']
  return_key 'parameter_values'
  action :get_parameters
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end

aws_ssm_parameter_store 'getParametersbypath' do
  path '/pathtest/'
  recursive true
  with_decryption true
  return_key 'path_values'
  action :get_parameters_by_path
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end
Get bucket name and retrieve file
aws_ssm_parameter_store 'get bucketname' do
  path 'bucketname'
  return_key 'bucketname'
  action :get
  aws_access_key node['aws_test']['key_id']
  aws_secret_access_key node['aws_test']['access_key']
end

aws_s3_file "/tmp/test.txt" do
  bucket lazy {node.run_state['bucketname']}
  remote_path "test.txt"
  sensitive true
  aws_access_key_id node[:custom_access_key]
  aws_secret_access_key node[:custom_secret_key]
end

aws_autoscale

autoscale can be used to attach and detach EC2 instances to/from an AutoScaling Group (ASG). Once the instance is attached autoscale allows one to move the instance into and out of standby mode. Standby mode temporarily takes the instance out of rotation so that maintenance can be performed.

Properties

Actions

Examples

aws_autoscaling 'attach_instance' do
  action :attach_instance
  asg_name 'Test'
end
aws_autoscaling 'enter_standby' do
  should_decrement_desired_capacity true
  action :enter_standby
end
 aws_autoscaling 'exit_standby' do
   action :exit_standby
 end
aws_autoscaling 'detach_instance' do
  should_decrement_desired_capacity true
  action :detach_instance
end

Contributors

This project exists thanks to all the people who contribute.

Backers

Thank you to all our backers!

https://opencollective.com/sous-chefs#backers

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website.

https://opencollective.com/sous-chefs/sponsor/0/website https://opencollective.com/sous-chefs/sponsor/1/website https://opencollective.com/sous-chefs/sponsor/2/website https://opencollective.com/sous-chefs/sponsor/3/website https://opencollective.com/sous-chefs/sponsor/4/website https://opencollective.com/sous-chefs/sponsor/5/website https://opencollective.com/sous-chefs/sponsor/6/website https://opencollective.com/sous-chefs/sponsor/7/website https://opencollective.com/sous-chefs/sponsor/8/website https://opencollective.com/sous-chefs/sponsor/9/website