18 add support for ec2 instance

[enekui]

Changes

• Refactored VPC networking stack. Now the customer don't need to interact with the networking design, it's all computed in runtime by Terraform using the cidrsubnet fuction in combination with other functions. This improvement also allows that availability zones to be computed by Terraform.
• Added the capability to deploy the EC2 instance in a public network and get public IP, this had a new chanllenge since the RDS and Redis are always deployed to private networks, to resolve this I added support for VPC endpoints that will allow this communication flow from the EC2 public network using VPC endpoints.
• Added support for SSMSessionManager as a connectivity solution to the EC2 instance, since when the instance is deployed to private networks SSH won't work.
• Added support to auto-generate ssh key in the cloud from local SSH key material.
• Added variable validation for the VPC cidr, and num of availablity zones to keep this under control.
• Updated README.md
• Added other small improvements to code.

Example of public EC2 instance main.tf

module "terraform-seqera-module" {
  source  = "github.com/seqeralabs/terraform-seqera-aws"

  region  = "eu-west-2"

  ## VPC
  vpc_name = "seqera-vpc"

  ## EC2 Instance
  create_ec2_instance = true
  create_ec2_instance_local_key_pair = true
  create_public_ec2_instance = true

  default_tags = {
    Environment = "development"
    ManagedBy   = "Terraform"
    Product     = "Seqera"
    CreatedBy   = "DevOps"
  }
}

output "database_url" {
  value = module.terraform-seqera-module.database_url
}

output "redis_url" {
  value = module.terraform-seqera-module.redis_url
}

Example of private EC2 instance

module "terraform-seqera-module" {
  source  = "github.com/seqeralabs/terraform-seqera-aws"

  region  = "eu-west-2"

  ## VPC
  vpc_name = "seqera-vpc"

  ## EC2 Instance
  create_ec2_instance = true
  enable_ec2_instance_session_manager_access = true

  default_tags = {
    Environment = "development"
    ManagedBy   = "Terraform"
    Product     = "Seqera"
    CreatedBy   = "DevOps"
  }
}

output "database_url" {
  value = module.terraform-seqera-module.database_url
}

output "redis_url" {
  value = module.terraform-seqera-module.redis_url
}

Example of public EC2 instance with secure SSM Session Manager access main.tf

module "terraform-seqera-module" {
  source  = "github.com/seqeralabs/terraform-seqera-aws"

  region  = "eu-west-2"

  ## VPC
  vpc_name = "seqera-vpc"

  ## EC2 Instance
  create_ec2_instance = true
  enable_ec2_instance_session_manager_access = true
  create_public_ec2_instance = true
  ec2_instance_security_group_ingress_rules_names = ["http-80-tcp", "https-443-tcp"]

  default_tags = {
    Environment = "development"
    ManagedBy   = "Terraform"
    Product     = "Seqera"
    CreatedBy   = "DevOps"
  }
}

output "database_url" {
  value = module.terraform-seqera-module.database_url
}

output "redis_url" {
  value = module.terraform-seqera-module.redis_url
}

Commit

f7e9df189943aaab2fcbc61bba91724146304428

[pditommaso]

Tried this. It looks like aws_profile is not supported any more

» terraform apply
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 4, in module "terraform-seqera-module":
│    4:   aws_profile = "deploy-seqera-platform"
│ 
│ An argument named "aws_profile" is not expected here.

Removing it, even worse

╷
│ Error: Invalid provider configuration
│ 
│ Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and configure
│ the provider's required arguments as described in the provider documentation.
│ 
╵
╷
│ Error: configuring Terraform AWS Provider: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, failed to resolve service endpoint, an AWS region is required, but was not found
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on <empty> line 0:
│   (source code not available)
│ 
╵

[enekui]

Tried this. It looks like aws_profile is not supported any more

» terraform apply

╷

│ Error: Unsupported argument

│ 

│   on main.tf line 4, in module "terraform-seqera-module":

│    4:   aws_profile = "deploy-seqera-platform"

│ 

│ An argument named "aws_profile" is not expected here.

Removing it, even worse

╷

│ Error: Invalid provider configuration

│ 

│ Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and configure

│ the provider's required arguments as described in the provider documentation.

│ 

╵

╷

│ Error: configuring Terraform AWS Provider: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, failed to resolve service endpoint, an AWS region is required, but was not found

│ 

│   with provider["registry.terraform.io/hashicorp/aws"],

│   on <empty> line 0:

│   (source code not available)

│ 

╵

Yes, I removed it in the last commit. It's not needed any more, since we are not using the command line cli.

[pditommaso]

Yes, I removed it in the last commit. It's not needed any more, since we are not using the command line cli

Then, how to specify the credentials to be used? I get the above error when omitting it

[enekui]

Mmm, I think you are mixing concepts here. The aws_profile variable was being used in the main.tf file only. It's independent from the profile parameter required by terraform in the provider configuration so, it only was removed from the main.tf and I believe you are removing it from the provider block or somewhere else.

[pditommaso]

Got it

[pditommaso]

Ok getting better

Apply complete! Resources: 58 added, 0 changed, 0 destroyed.

Outputs:

database_url = "seqera-db.cdvfyscyteue.eu-central-1.rds.amazonaws.com"
redis_url = "seqera-redis.3e3qgc.ng.0001.euc1.cache.amazonaws.com"

The output should include the Ec2 instance address. How to SSH to it?

[enekui]

Ok getting better

Apply complete! Resources: 58 added, 0 changed, 0 destroyed.

Outputs:

database_url = "seqera-db.cdvfyscyteue.eu-central-1.rds.amazonaws.com"
redis_url = "seqera-redis.3e3qgc.ng.0001.euc1.cache.amazonaws.com"

The output should include the Ec2 instance address. How to SSH to it?

Depends of which kind of EC2 instance you deployed. Share your main.tf so I can see what did you deployed. Note that SSH will only work with public EC2 instances.

[pditommaso]

Small thing, let's rename create_public_ec2_instance to create_ec2_public_instance

[enekui]

Small thing, let's rename create_public_ec2_instance to create_ec2_public_instance

Good point! Sure!

[enekui]

Did a full deployment of the EKS cluster after these changes and we get all working as respected. ConfigMap, Namespace, Secrets, IAM role, policy, etc.

DB and Redis connectivity already tested too.

[pditommaso]

I'm getting this error

╷
│ Error: Failed to get existing workspaces: Unable to list objects in S3 bucket "seqera-platform-tf-test-de": operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: MDK5V8JXQQS58FMV, HostID: u2/RMoW/6mqc0HZdvolKgw1FBtoUu8EiRNh0k2l+5fgsfl2ak/dcrlFl70O/v/MzsISzjr5T5oo=, api error AccessDenied: Access Denied
│ 
│ 
╵

in the main.nf I have

terraform {
  backend "s3" {
    ## the bucket must exist
    bucket  = "seqera-platform-tf-test-de"
    key     = "state"
    region  = "eu-central-1"
    profile = "deploy-seqera-platform"
  }
}

# Configure the AWS Provider
provider "aws" {
  region  = "eu-central-1"
  profile = "deploy-seqera-platform"
}

and using the CLI it looks look

» aws --profile deploy-seqera-platform s3 ls s3://seqera-platform-tf-test-de/
2023-11-20 17:13:00        179 state

[pditommaso]

it was the usual problem with AWS_ env variables

[pditommaso]

Not sure what you tested, I got this when using this main:

╷
│ Error: Unsupported argument
│ 
│   on main.tf line 6, in module "terraform-seqera-aws":
│    6:   aws_profile = "deploy-seqera-platform"
│ 
│ An argument named "aws_profile" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 17, in module "terraform-seqera-aws":
│   17:   azs                 = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]
│ 
│ An argument named "azs" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 20, in module "terraform-seqera-aws":
│   20:   database_subnets    = ["10.0.104.0/24", "10.0.105.0/24", "10.0.106.0/24"]
│ 
│ An argument named "database_subnets" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 21, in module "terraform-seqera-aws":
│   21:   elasticache_subnets = ["10.0.107.0/24", "10.0.108.0/24", "10.0.109.0/24"]
│ 
│ An argument named "elasticache_subnets" is not expected here.
╵
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 22, in module "terraform-seqera-aws":
│   22:   intra_subnets       = ["10.0.110.0/24", "10.0.111.0/24", "10.0.112.0/24"]
│ 
│ An argument named "intra_subnets" is not expected here.
╵

[enekui]

Not sure what you tested, I got this when using this main:

╷

│ Error: Unsupported argument

│ 

│   on main.tf line 6, in module "terraform-seqera-aws":

│    6:   aws_profile = "deploy-seqera-platform"

│ 

│ An argument named "aws_profile" is not expected here.

╵

╷

│ Error: Unsupported argument

│ 

│   on main.tf line 17, in module "terraform-seqera-aws":

│   17:   azs                 = ["eu-central-1a", "eu-central-1b", "eu-central-1c"]

│ 

│ An argument named "azs" is not expected here.

╵

╷

│ Error: Unsupported argument

│ 

│   on main.tf line 20, in module "terraform-seqera-aws":

│   20:   database_subnets    = ["10.0.104.0/24", "10.0.105.0/24", "10.0.106.0/24"]

│ 

│ An argument named "database_subnets" is not expected here.

╵

╷

│ Error: Unsupported argument

│ 

│   on main.tf line 21, in module "terraform-seqera-aws":

│   21:   elasticache_subnets = ["10.0.107.0/24", "10.0.108.0/24", "10.0.109.0/24"]

│ 

│ An argument named "elasticache_subnets" is not expected here.

╵

╷

│ Error: Unsupported argument

│ 

│   on main.tf line 22, in module "terraform-seqera-aws":

│   22:   intra_subnets       = ["10.0.110.0/24", "10.0.111.0/24", "10.0.112.0/24"]

│ 

│ An argument named "intra_subnets" is not expected here.

╵

It's tested, but remember we removed that variable. So, you need to remove it from the main.tf We already talked about this.