search

clientIO / appmixer-module-aws

Appmixer Infrastructure-as-Code Terraform module for AWS.
Apache License 2.0
0 stars 0 forks source link

readme

AWS Appmixer Terraform module

Terraform validate pre-commit

Description

A Terraform module to provision appmixer application.

Configuration

Autoscaling configuration

Autoscaling is configured using ecs_autoscaling_config variable which by default defines two capacity providers used while scaling the cluster.

[!IMPORTANT] Correct Instance types need to be set. The size of EC2 network interfaces (ENIs) for each instance type is restricted and this limitation is directly tied to the number of containers that can be executed on the instance. To avoid this limitation, consider using instance types that allow so-called ENI Trunking meaning EC2 can attach more network interfaces (see more AWS Docs) Also ENI Trunking must be allowed in AWS account (AWS ECS -> Account Settings -> AWSVPC Trunking -> Enable)

Managed services

RabbitMQ, Opensearch (Elasticsearch), ElasticCache (Redis), DocumentDB(Mongo) are managed by AWS and can be configured through variables:

Each service is running with minimal configuration, in production they might need some resizing.

[!NOTE] Within this module, managed services can be created separately and incorporated using the e.g. external_elasticache variable, for instance. It is crucial in this setup to verify the correct configuration of all network and security policies.

User initialization

Admin user is initialized through automated task running in ECS. Variable init_user needs to be set.

Deployment & Upgrade

Examples

Pricing

AWS Pricing estimation for each environemnt. Pricing can vary based on real traffic.

Simple VPN

You can use sshuttle to connect to private VPC network.

  1. Run EC2 with configured SSH keys in your public subnet
  2. Install sshuttle
  3. Connect through sshuttle to your running EC2 (sshuttle -r <SSH_CONFIG> 0.0.0.0/0 -vv)
  4. Your network traffic should be routed through EC2, you should be able to see managed AWS services running in private subnets.

Architecture

arch-diagram

Requirements

Name Version
terraform >= 1.6.6
aws >= 5.32.1
http 3.4.1
random >= 3.6.0

Modules

Name Source Version
alb terraform-aws-modules/alb/aws ~> 9.0
autoscaling terraform-aws-modules/autoscaling/aws ~> 6.5
autoscaling_sg terraform-aws-modules/security-group/aws ~> 5.0
document_db_ssm_password cloudposse/ssm-parameter-store/aws 0.11.0
document_db_ssm_username cloudposse/ssm-parameter-store/aws 0.11.0
documentdb_cluster cloudposse/documentdb-cluster/aws 0.24.0
ecs_cluster terraform-aws-modules/ecs/aws//modules/cluster 5.7.4
ecs_service_backoffice terraform-aws-modules/ecs/aws//modules/service 5.7.4
ecs_service_engine terraform-aws-modules/ecs/aws//modules/service 5.7.4
ecs_service_frontend terraform-aws-modules/ecs/aws//modules/service 5.7.4
ecs_service_logstash terraform-aws-modules/ecs/aws//modules/service 5.7.4
ecs_service_quota terraform-aws-modules/ecs/aws//modules/service 5.7.4
elasticache cloudposse/elasticache-redis/aws 1.0.0
elasticache_ssm_password cloudposse/ssm-parameter-store/aws 0.11.0
elasticsearch cloudposse/elasticsearch/aws 0.46.0
elasticsearch_ssm_password cloudposse/ssm-parameter-store/aws 0.11.0
elasticsearch_ssm_username cloudposse/ssm-parameter-store/aws 0.11.0
label cloudposse/label/null 0.25.0
rabbit_mq cloudposse/mq-broker/aws 3.1.0
s3_bucket cloudposse/s3-bucket/aws 4.0.1
services_configuration_merge cloudposse/config/yaml//modules/deepmerge 1.0.2
sg_document_db_rules cloudposse/security-group/aws 2.2.0
sg_user_init cloudposse/security-group/aws 2.2.0
sq_elasticsearch cloudposse/security-group/aws 2.2.0
subnets cloudposse/dynamic-subnets/aws 2.4.1
vpc cloudposse/vpc/aws 2.1.1

Resources

Name Type
aws_acm_certificate.alb resource
aws_acm_certificate_validation.this resource
aws_cloudwatch_log_group.ecs_mongo_init_user resource
aws_ecs_task_definition.this resource
aws_elasticsearch_domain_policy.this resource
aws_route53_record.alb resource
aws_route53_record.cert_alb resource
aws_service_discovery_http_namespace.appmixer resource
random_password.elasticsearch_password resource
random_password.rabbit_mq resource
random_password.redis_password resource
random_pet.document_db_username resource
random_pet.elasticsearch_username resource
aws_ecs_task_execution.run data source
aws_region.current data source
aws_ssm_parameter.ecs_optimized_ami data source
aws_vpc.external data source
http_http.init_user data source

Inputs

Name Description Type Default Required
root_dns_name Root DNS name, must be applicable to route53 zone (zone_id) string n/a yes
additional_security_group_rules Additional security group rules added to security group rules of all resources, see more Terraform docs 
list(object({
    type              = string
    from_port         = number
    to_port           = number
    protocol          = string
    cidr_blocks       = optional(list(string))
    ipv6_cidr_blocks  = optional(list(string))
    security_group_id = optional(string)
  }))
 [] no
alb_ingress_security_group_rules Application Load Balancer security group ingress rules 
map(object({
    ip_protocol                  = string
    from_port                    = optional(number)
    to_port                      = optional(number)
    referenced_security_group_id = optional(string)
    description                  = optional(string)
    cidr_ipv4                    = optional(string)
    cidr_ipv6                    = optional(string)
  }))
 
{
  "all_http": {
    "cidr_ipv4": "0.0.0.0/0",
    "from_port": 80,
    "ip_protocol": "tcp",
    "to_port": 80
  },
  "all_https": {
    "cidr_ipv4": "0.0.0.0/0",
    "from_port": 443,
    "ip_protocol": "tcp",
    "to_port": 443
  }
}
 no
attributes Additional attributes (e.g. 1) list(string) [] no
availability_zones List of availability zones list(string) 
[
  "eu-central-1a",
  "eu-central-1b",
  "eu-central-1c"
]
 no
certificate_arn Certificate ARN, if not set, certificate will be automatically created using '*.<root_dns_name>' string null no
document_db DocumentDB configuration object 
object({
    cluster_size   = optional(number, 1)
    cluster_family = optional(string, "docdb5.0")
    instance_class = optional(string, "db.t4g.medium")
    engine_version = optional(string, "5.0.0")
    cluster_parameters = optional(list(object({
      apply_method = string
      name         = string
      value        = string
    })), [])
  })
 {} no
ecs_autoscaling_config n/a any 
{
  "on_demand": {
    "capacity_provider": {
      "default_capacity_provider_strategy": {
        "base": 1,
        "weight": 10
      },
      "maximum_scaling_step_size": 5,
      "minimum_scaling_step_size": 1,
      "target_capacity": 100
    },
    "instance_type": "m5.large",
    "max_size": 6,
    "min_size": 1,
    "mixed_instances_policy": {
      "instances_distribution": {
        "on_demand_allocation_strategy": "prioritized",
        "on_demand_base_capacity": 1,
        "on_demand_percentage_above_base_capacity": 100,
        "spot_allocation_strategy": "lowest-price"
      },
      "override": [
        {
          "instance_type": "m5.large",
          "weighted_capacity": "1"
        },
        {
          "instance_type": "c5.large",
          "weighted_capacity": "1"
        }
      ]
    },
    "use_mixed_instances_policy": true
  },
  "spot": {
    "capacity_provider": {
      "default_capacity_provider_strategy": {
        "base": 0,
        "weight": 80
      },
      "maximum_scaling_step_size": 5,
      "minimum_scaling_step_size": 1,
      "target_capacity": 100
    },
    "instance_type": "m5.large",
    "max_size": 6,
    "min_size": 1,
    "mixed_instances_policy": {
      "instances_distribution": {
        "on_demand_allocation_strategy": "prioritized",
        "on_demand_base_capacity": 0,
        "on_demand_percentage_above_base_capacity": 0,
        "spot_allocation_strategy": "lowest-price"
      },
      "override": [
        {
          "instance_type": "m5.large",
          "weighted_capacity": "1"
        },
        {
          "instance_type": "c5.large",
          "weighted_capacity": "1"
        }
      ]
    },
    "use_mixed_instances_policy": true
  }
}
 no
ecs_cluster_config Cluster configuration object execute_command_configuration, see more terraform docs any 
{
  "log_configuration": {
    "cloud_watch_log_group_name": "/aws/ecs/aws-ec2"
  },
  "logging": "OVERRIDE"
}
 no
ecs_common_service_config ECS service configuration:
- ordered_placement_strategy defines how tasks are placed on instances, see more AWS docs or Terraform docs
- force_new_deployment force service redeployment
- wait_for_steady_state terraform apply waits for service to reach steady state, see more Terraform docs		 
object({
    ordered_placement_strategy = optional(list(object({
      type  = string
      field = string
      })), [{
      type  = "binpack"
      field = "cpu"
    }])
    force_new_deployment     = optional(bool, false)
    wait_for_steady_state    = optional(bool, true)
    autoscaling_min_capacity = optional(number, 1)
    autoscaling_max_capacity = optional(number, 10)
    deployment_circuit_breaker = optional(object({
      enable   = bool
      rollback = bool
      }), {
      enable   = true
      rollback = true
    })
  })
 {} no
ecs_per_service_config Configuration per service, overrides 'ecs_common_service_config'
Example:
{
      engine = {
        image = "registry.appmixer.com/appmixer-engine:5.2.0-nocomp"
        url   = "api.ecs.appmixer.co"
        env = {
          EXAMPLE_ENV = "example"
        }
        cpu          = 512
        memory       = 1024
        health_check = {}
        entrypoint = [ "node", "gridd.js", "--http", "--emails" ]
        autoscaling_min_capacity = 1
        autoscaling_max_capacity = 10
        force_new_deployment = true
        wait_for_steady_state = true
        ordered_placement_strategy = [{
          type  = "binpack"
          field = "cpu"
        }]
      }
      quota = {...}
      frontend = {...}
      backoffice = {...}
      logstash = {...}
      }
    }
 any {} no
ecs_registry_auth_data Docker registry credentials, base64 encoded string string "" no
elasticache Elastic module configuration object 
object({
    cluster_size               = optional(number, 1)
    instance_type              = optional(string, "cache.t3.micro")
    engine_version             = optional(string, "6.2")
    family                     = optional(string, "redis6.x")
    at_rest_encryption_enabled = optional(bool, true)
    transit_encryption_enabled = optional(bool, true)
    automatic_failover_enabled = optional(bool, false)
    parameter = optional(list(object({
      name  = string
      value = string
    })), [])
  })
 {} no
elasticsearch Elasticsearch module configuration object 
object({
    elasticsearch_version           = optional(string, "OpenSearch_2.7")
    instance_type                   = optional(string, "t3.medium.elasticsearch")
    instance_count                  = optional(number, 1)
    ebs_volume_size                 = optional(number, 20)
    encrypt_at_rest_enabled         = optional(bool, true)
    advanced_options                = optional(map(string), null)
    node_to_node_encryption_enabled = optional(bool, true)
  })
 
{
  "advanced_options": {
    "rest.action.multi.allow_explicit_index": "true"
  }
}
 no
enable_deletion_protection Enable deletion protection for all managed resources, if true, resources can't be deleted if not explicitly set to false bool true no
environment Environment, e.g. 'prod', 'staging', 'dev', 'pre-prod', 'UAT' string "dev" no
external_documentdb Connection string to DocumentDB, if not set, DocumentDB will be automatically created string null no
external_elasticsearch Connection object to Elasticsearch, if not set, Elasticsearch will be automatically created 
object({
    url      = string
    username = string
    password = string
  })
 null no
external_rabbitmq Connection object to RabbitMQ, if not set, RabbitMQ will be automatically created 
object({
    url      = string
    username = string
    password = string
    port     = number
  })
 null no
external_redis Connection string to Redis, if not set, Redis will be automatically created string null no
external_vpc VPC configuration, if not set, new VPC will be created 
object({
    vpc_id             = string
    public_subnet_ids  = list(string)
    private_subnet_ids = list(string)
  })
 null no
init_user Initial user created in appmixer. Creation through appmixer API and by setting up admin scope in documentdb directly 
object({
    email             = optional(string, "")
    username          = optional(string, "")
    password          = optional(string, "")
    max_retry_minutes = optional(number, 45)
  })
 null no
name Solution name, e.g. 'appmixer' string "appmixer" no
namespace Namespace, which could be your organization name or abbreviation, e.g. 'eg' or 'cp' string "cio" no
rabbitmq RabbitMQ module configuration object 
object({
    auto_minor_version_upgrade = optional(bool, true)
    deployment_mode            = optional(string, "SINGLE_INSTANCE")
    engine_version             = optional(string, "3.8.34")
    host_instance_type         = optional(string, "mq.t3.micro")
    audit_log_enabled          = optional(bool, false)
    general_log_enabled        = optional(bool, true)
    encryption_enabled         = optional(bool, true)
    use_aws_owned_key          = optional(bool, false)
    publicly_accessible        = optional(bool, false)
  })
 {} no
s3_config Configuration for S3 bucket 
object({
    versioning_enabled = optional(bool, false)
    logging            = optional(list(object({ bucket_name = string, prefix = string })), [])
  })
 {} no
stage Stage, e.g. 'prod', 'staging', 'dev' string "" no
tags Additional tags (e.g. map('BusinessUnit','XYZ') map(string) {} no
vpc_config VPC configuration, ignored if external_vpc is set 
object({
    ipv4_primary_cidr_block = string
    availability_zones      = list(string)
  })
 
{
  "availability_zones": [
    "eu-central-1a",
    "eu-central-1b",
    "eu-central-1c"
  ],
  "ipv4_primary_cidr_block": "10.0.0.0/16"
}
 no
zone_id Route53 DNS zone ID, if not set AWS route53 will be not used string null no

Outputs

Name Description
alb_dns_name DNS name of the ALB
managed_services Managed services configuration containing endpoints and name of SSM parameters with credentials
services_urls URLs of the running services
vpc_config VPC configuration contaning VPC ID, CIDR block and subnets

Contributing and reporting issues

Feel free to create an issue in this repository if you have questions, suggestions or feature requests.

Validation, linters and pull-requests

We want to provide high quality code and modules. For this reason we are using several pre-commit hooks and GitHub Actions workflows. A pull-request to the main branch will trigger these validations and lints automatically. Please check your code before you will create pull-requests. See pre-commit documentation and GitHub Actions documentation for further details.

License

License

See LICENSE for full details.

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.