Closed DingosGotMyBaby closed 1 year ago
lukaevet
commented
1 year ago Try to write your Cloudformation code in a different way. Probably the conditions are what confuses this convertor. In my case my .yml file looks like this: `SecurityGroupIngress:
shadycuz
commented
1 year ago @DingosGotMyBaby Really cool project you have!
The tool has a debug flag. It might show more information on the problem. I will try and run it myself when I get time.
If your template is valid Cloudformation then the bug is in this converter.
As @lukaevet pointed out, I think the issue is with the !If conditions here:
Ec2Sg:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub "${AWS::StackName}-ec2"
GroupDescription: !Sub "${AWS::StackName}-ec2"
SecurityGroupIngress:
- !If
- IPv4AddressProvided
- FromPort: 22
ToPort: 22
IpProtocol: tcp
CidrIp: !Sub "${YourIPv4}/32"
- !Ref 'AWS::NoValue'
- !If
- IPv6AddressProvided
- FromPort: 22
ToPort: 22
IpProtocol: tcp
CidrIpv6: !Sub "${YourIPv6}/128"
- !Ref 'AWS::NoValue'
- FromPort: 25565
ToPort: 25565
IpProtocol: tcp
CidrIp: 0.0.0.0/0
VpcId: !Ref VpcIt probably has something to do with !Ref 'AWS::NoValue'. It could be a while before I fix this 😢 , so for a workaround for now you could remove the conditionals and hopefully 🤞 it will finish the conversion. Than in Terraform you can just put the conditions back in.
Ec2Sg:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub "${AWS::StackName}-ec2"
GroupDescription: !Sub "${AWS::StackName}-ec2"
SecurityGroupIngress:
- FromPort: 22
ToPort: 22
IpProtocol: tcp
CidrIp: !Sub "${YourIPv4}/32"
- FromPort: 22
ToPort: 22
IpProtocol: tcp
CidrIpv6: !Sub "${YourIPv6}/128"
- FromPort: 25565
ToPort: 25565
IpProtocol: tcp
CidrIp: 0.0.0.0/0
VpcId: !Ref Vpc@DingosGotMyBaby I just tried the template you linked to and did not get your error. Instead, I got a different error.
raise ValueError(f"{key} not allowed to be nested in {prev_func}.")
ValueError: Fn::Cidr not allowed to be nested in Fn::Select.This is expected because the AWS documentation for !Select states only a small set of functions can be nested inside of it. However... we have seen on multiple occasions that the AWS documentation is WRONG. So as long as you are positive that this code works, we can update the converter to allow that nesting.
But I removed that and eventually got the error that you created this issue about. I did some digging and I kinda understand what is going on.
The way cf2tf works, is it will first convert all the Cloudformation functions to Terraform functions. After that it will convert all the property names from CF to TF attributes.
The problem here is that:
SecurityGroupIngress:
- !If
- IPv4AddressProvided
- FromPort: 22
ToPort: 22
IpProtocol: tcp
CidrIp: !Sub "${YourIPv4}/32"
- !Ref 'AWS::NoValue'is being converted to:
ingress = local.IPv4AddressProvided ? {
FromPort = 22
ToPort = 22
IpProtocol = "tcp"
CidrIp = "${var.your_i_pv4}/32"
} : nullThe issues are:
cf2tf understands this as a Terraform documentation Section called ingress with a value of type Dict. It is expecting the value to have the keys FromPort, ToPort etc. that need to be converted from CF to TF. This a pretty valid assumption but fails when the actual value is wrapped in a function as in your code example. The only fix I can see would be to first convert the properties to attributes and then convert the functions. But I'm guessing there is a reason I do it in this order, but I will try it if I get time. I'm not sure this can actually be "fixed". Your Cloudformation logic is perfectly fine but its not really compatible with how it would be expressed in Terraform. Which is kinda exciting? This is the first known case. I will continue to think of ways we can get closer to being able to convert this template 100%, but for now I think the best option would be to find a way to not throw such a harsh error and instead comment out the offending line and just move on with the conversion?
For future me, the first step is going to be handling the conversion of Cloudformation List of Maps to Terraform Nested Blocks as outlined here:
The next step will be to figure out what to do in the case that an entire section of properties is going to get wrapped in a function conversion.
For now I'm going to try and make sure that we don't panic and instead continue on with the template conversion.
DingosGotMyBaby
commented
1 year ago Thank for looking into this, Cloudformation created all the resources correctly so it looks like it's a valid template.
It wasn't a clean convert so I have been manually rewriting a lot of the generated terraform but it gave a good boiler plate for something I couldn't understand properly at first.
Thank you for working on a great project!
shadycuz
commented
1 year ago @DingosGotMyBaby I have a patch in place locally that doesn't panic on the error. Here is the converted Terraform.
data "aws_region" "current" {}
locals {
mappings = {
ServerState = {
Running = {
DesiredCapacity = 1
}
Stopped = {
DesiredCapacity = 0
}
}
}
MinecraftTypeTagProvided = !var.minecraft_type_tag == ""
AdminPlayerNamesProvided = !var.admin_player_names == ""
DifficultyProvided = !var.difficulty == ""
WhitelistProvided = !var.whitelist == ""
MinecraftVersionProvided = !var.minecraft_version == ""
EntryPointProvided = !join("", var.entry_point) == ""
CommandProvided = !join("", var.command) == ""
LogGroupNameProvided = !var.log_group_name == ""
LogStreamPrefixProvided = !var.log_stream_prefix == ""
KeyPairNameProvided = !var.key_pair_name == ""
IPv4AddressProvided = !var.your_i_pv4 == ""
IPv6AddressProvided = !var.your_i_pv6 == ""
DnsConfigEnabled = alltrue([
!var.hosted_zone_id == "",
!var.record_name == ""
])
SpotPriceProvided = !var.spot_price == ""
MemoryProvided = !var.memory == ""
SeedProvided = !var.seed == ""
MaxPlayersProvided = !var.max_players == -1
ViewDistanceProvided = !var.view_distance == -1
GameModeProvided = !var.game_mode == ""
LevelTypeProvided = !var.level_type == ""
EnableRollingLogsProvided = !var.enable_rolling_logs == ""
TimezoneProvided = !var.timezone == ""
stack_name = "minecraft"
}
variable "ecsami" {
description = "AWS ECS AMI ID"
type = string
default = "/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id"
}
variable "server_state" {
description = "Running: A spot instance will launch shortly after setting this parameter; your Minecraft server should start within 5-10 minutes of changing this parameter (once UPDATE_IN_PROGRESS becomes UPDATE_COMPLETE). Stopped: Your spot instance (and thus Minecraft container) will be terminated shortly after setting this parameter."
type = string
default = "Running"
}
variable "instance_type" {
description = "t3.medium is a good cost-effective instance, 2 vCPUs and 3.75 GB of RAM with moderate network performance. Change at your discretion. https://aws.amazon.com/ec2/instance-types/."
type = string
default = "t3.medium"
}
variable "spot_price" {
description = "A t3.medium shouldn't cost much more than a cent per hour. Note: Leave this blank to use on-demand pricing."
type = string
default = "0.05"
}
variable "container_insights" {
description = "Enable/Disable ECS Container Insights for ECS Cluster"
type = string
default = "disabled"
}
variable "entry_point" {
description = "Task entrypoint (Optional - image default is script /start)"
type = string
}
variable "command" {
description = "Task command (Optional - image default is empty)"
type = string
}
variable "log_group_name" {
description = "(Optional - An empty value disables this feature)"
type = string
}
variable "log_group_retention_in_days" {
description = "(Log retention in days)"
type = string
default = 7
}
variable "log_stream_prefix" {
description = "(Optional)"
type = string
default = "minecraft-server"
}
variable "key_pair_name" {
description = "(Optional - An empty value disables this feature)"
type = string
}
variable "your_i_pv4" {
description = "(Optional - An empty value disables this feature)"
type = string
}
variable "your_i_pv6" {
description = "(Optional - An empty value disables this feature)"
type = string
}
variable "hosted_zone_id" {
description = "(Optional - An empty value disables this feature) If you have a hosted zone in Route 53 and wish to set a DNS record whenever your Minecraft instance starts, supply the hosted zone ID here."
type = string
}
variable "record_name" {
description = "(Optional - An empty value disables this feature) If you have a hosted zone in Route 53 and wish to set a DNS record whenever your Minecraft instance starts, supply the name of the record here (e.g. minecraft.mydomain.com)."
type = string
}
variable "minecraft_image_tag" {
description = "Java version (Examples include latest, adopt13, openj9, etc) Refer to tag descriptions available here: https://github.com/itzg/docker-minecraft-server)"
type = string
default = "latest"
}
variable "minecraft_type_tag" {
description = "(Examples include SPIGOT, BUKKIT, TUINITY, etc) Refer to tag descriptions available here: https://github.com/itzg/docker-minecraft-server)"
type = string
}
variable "admin_player_names" {
description = "Op/Administrator Players"
type = string
}
variable "difficulty" {
description = "The game's difficulty"
type = string
default = "normal"
}
variable "whitelist" {
description = "Usernames of your friends"
type = string
}
variable "minecraft_version" {
description = "Server minecraft version"
type = string
}
variable "memory" {
description = "How much Memory to allocate for the JVM"
type = string
default = "1G"
}
variable "seed" {
description = "The seed used to generate the world"
type = string
}
variable "max_players" {
description = "Max number of players that can connect simultaneously (default 20)"
type = string
default = -1
}
variable "view_distance" {
description = "Max view radius (in chunks) the server will send to the client (default 10)"
type = string
default = -1
}
variable "game_mode" {
description = "Options: creative, survival (default), adventure, spectator (v1.8+)"
type = string
default = "survival"
}
variable "level_type" {
description = "Options: DEFAULT, FLAT, LARGEBIOMES, AMPLIFIED, CUSTOMIZED, BUFFET, BIOMESOP (v1.12-), BIOMESOPLENTY (v1.15+)"
type = string
default = "DEFAULT"
}
variable "enable_rolling_logs" {
description = "By default the log file will grow without limit. Set to true to use a rolling log strategy."
type = string
default = False
}
variable "timezone" {
description = "Change the server's timezone. Use the canonical name of the format: Area/Location (e.g. America/New_York)"
type = string
}
resource "aws_vpc" "vpc" {
cidr_block = "10.100.0.0/26"
enable_dns_support = True
enable_dns_hostnames = True
}
resource "aws_subnet" "subnet_a" {
availability_zone = element(data.aws_availability_zones.available.names, 0)
cidr_block = cidrsubnets(""10.100.0.0/26"", 2, 2, 2, 2)
vpc_id = aws_vpc.vpc.arn
}
resource "aws_route_table_association" "subnet_a_route" {
route_table_id = aws_route_table.route_table.id
subnet_id = aws_subnet.subnet_a.id
}
resource "aws_route_table_association" "subnet_b_route" {
route_table_id = aws_route_table.route_table.id
subnet_id = aws_subnet.subnet_b.id
}
resource "aws_subnet" "subnet_b" {
availability_zone = element(data.aws_availability_zones.available.names, 1)
cidr_block = cidrsubnets(""10.100.0.0/26"", 2, 2, 2, 2)
vpc_id = aws_vpc.vpc.arn
}
resource "aws_internet_gateway" "internet_gateway" {}
resource "aws_ec2_transit_gateway_vpc_attachment" "internet_gateway_attachment" {
vpc_id = aws_vpc.vpc.arn
}
resource "aws_route_table" "route_table" {
vpc_id = aws_vpc.vpc.arn
}
resource "aws_route" "route" {
destination_cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.internet_gateway.id
route_table_id = aws_route_table.route_table.id
}
resource "aws_efs_file_system" "efs" {}
resource "aws_efs_mount_target" "mount_a" {
file_system_id = aws_efs_file_system.efs.arn
security_groups = [
aws_security_group.efs_sg.arn
]
subnet_id = aws_subnet.subnet_a.id
}
resource "aws_efs_mount_target" "mount_b" {
file_system_id = aws_efs_file_system.efs.arn
security_groups = [
aws_security_group.efs_sg.arn
]
subnet_id = aws_subnet.subnet_b.id
}
resource "aws_security_group" "efs_sg" {
name = "${local.stack_name}-efs"
description = "${local.stack_name}-efs"
ingress = [
{
from_port = 2049
to_port = 2049
protocol = "tcp"
security_groups = aws_security_group.ec2_sg.arn
}
]
vpc_id = aws_vpc.vpc.arn
}
resource "aws_security_group" "ec2_sg" {
name = "${local.stack_name}-ec2"
description = "${local.stack_name}-ec2"
ingress = [
// local.IPv4AddressProvided ? {
// FromPort = 22
// ToPort = 22
// IpProtocol = "tcp"
// CidrIp = "${var.your_i_pv4}/32"
// } : null,
// local.IPv4AddressProvided ? {
// FromPort = 22
// ToPort = 22
// IpProtocol = "tcp"
// CidrIp = "${var.your_i_pv4}/32"
// } : null,
// local.IPv6AddressProvided ? {
// FromPort = 22
// ToPort = 22
// IpProtocol = "tcp"
// CidrIpv6 = "${var.your_i_pv6}/128"
// } : null,
// local.IPv6AddressProvided ? {
// FromPort = 22
// ToPort = 22
// IpProtocol = "tcp"
// CidrIpv6 = "${var.your_i_pv6}/128"
// } : null,
{
from_port = 25565
to_port = 25565
protocol = "tcp"
cidr_blocks = "0.0.0.0/0"
}
]
vpc_id = aws_vpc.vpc.arn
}
resource "aws_launch_configuration" "launch_configuration" {
associate_public_ip_address = True
iam_instance_profile = aws_iam_instance_profile.instance_profile.arn
image_id = var.ecsami
instance_type = var.instance_type
key_name = local.KeyPairNameProvided ? var.key_pair_name : null
security_groups = [
aws_security_group.ec2_sg.arn
]
spot_price = local.SpotPriceProvided ? var.spot_price : null
user_data = base64encode("#!/bin/bash -xe
echo ECS_CLUSTER=${aws_ecs_cluster.ecs_cluster.arn} >> /etc/ecs/ecs.config
yum install -y amazon-efs-utils
mkdir /opt/minecraft
mount -t efs ${aws_efs_file_system.efs.arn}:/ /opt/minecraft
chown 845:845 /opt/minecraft
")
}
resource "aws_neptune_subnet_group" "auto_scaling_group" {
name = aws_launch_configuration.launch_configuration.id
// CF Property(DesiredCapacity) = local.mappings["ServerState"][var.server_state]["DesiredCapacity"]
// CF Property(NewInstancesProtectedFromScaleIn) = True
// CF Property(MaxSize) = 1
// CF Property(MinSize) = 0
// CF Property(VPCZoneIdentifier) = [
// aws_subnet.subnet_a.id,
// aws_subnet.subnet_b.id
// ]
}
resource "aws_iam_role" "instance_role" {
assume_role_policy = {
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Service = [
"ec2.amazonaws.com"
]
}
Action = [
"sts:AssumeRole"
]
}
]
}
managed_policy_arns = [
"arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
]
}
resource "aws_iam_instance_profile" "instance_profile" {
role = [
aws_iam_role.instance_role.arn
]
}
resource "aws_ecs_cluster" "ecs_cluster" {
name = "${local.stack_name}-cluster"
setting = [
{
name = "containerInsights"
value = var.container_insights
}
]
}
resource "aws_ecs_capacity_provider" "ecs_capacity_provider" {
auto_scaling_group_provider = {
AutoScalingGroupArn = aws_neptune_subnet_group.auto_scaling_group.id
ManagedScaling = {
MaximumScalingStepSize = 1
MinimumScalingStepSize = 1
Status = "ENABLED"
TargetCapacity = 100
}
ManagedTerminationProtection = "ENABLED"
}
}
resource "aws_ecs_cluster" "ecs_cluster_capacity_provider_association" {
// CF Property(Cluster) = aws_ecs_cluster.ecs_cluster.arn
capacity_providers = [
aws_ecs_capacity_provider.ecs_capacity_provider.arn
]
default_capacity_provider_strategy = [
{
CapacityProvider = aws_ecs_capacity_provider.ecs_capacity_provider.arn
Weight = 1
}
]
}
resource "aws_ecs_service" "ecs_service" {
cluster = aws_ecs_cluster.ecs_cluster.arn
desired_count = local.mappings["ServerState"][var.server_state]["DesiredCapacity"]
name = "${local.stack_name}-ecs-service"
task_definition = aws_ecs_task_definition.ecs_task.arn
capacity_provider_strategy = [
{
CapacityProvider = aws_ecs_capacity_provider.ecs_capacity_provider.arn
Weight = 1
Base = 0
}
]
// CF Property(DeploymentConfiguration) = {
// MaximumPercent = 100
// MinimumHealthyPercent = 0
// }
}
resource "aws_ecs_task_definition" "ecs_task" {
volume = [
{
host_path = {
SourcePath = "/opt/minecraft"
}
name = "minecraft"
}
]
network_mode = "bridge"
container_definitions = [
{
Name = "minecraft"
MemoryReservation = 1024
Image = "itzg/minecraft-server:${var.minecraft_image_tag}"
EntryPoint = local.EntryPointProvided ? var.entry_point : null
Command = local.CommandProvided ? var.command : null
PortMappings = [
{
ContainerPort = 25565
HostPort = 25565
Protocol = "tcp"
}
]
LogConfiguration = local.LogGroupNameProvided ? {
LogDriver = "awslogs"
Options = {
awslogs-group = aws_scheduler_schedule_group.cloud_watch_log_group[0].id
awslogs-stream-prefix = local.LogStreamPrefixProvided ? "${var.log_stream_prefix}" : null
awslogs-region = data.aws_region.current.name
awslogs-create-group = True
}
} : null
MountPoints = [
{
ContainerPath = "/data"
SourceVolume = "minecraft"
ReadOnly = False
}
]
Environment = [
{
Name = "EULA"
Value = "TRUE"
},
local.MinecraftTypeTagProvided ? {
Name = "TYPE"
Value = "${var.minecraft_type_tag}"
} : null,
local.AdminPlayerNamesProvided ? {
Name = "OPS"
Value = "${var.admin_player_names}"
} : null,
local.DifficultyProvided ? {
Name = "DIFFICULTY"
Value = "${var.difficulty}"
} : null,
local.WhitelistProvided ? {
Name = "WHITELIST"
Value = "${var.whitelist}"
} : null,
local.MinecraftVersionProvided ? {
Name = "VERSION"
Value = "${var.minecraft_version}"
} : null,
local.MemoryProvided ? {
Name = "MEMORY"
Value = "${var.memory}"
} : null,
local.SeedProvided ? {
Name = "SEED"
Value = "${var.seed}"
} : null,
local.MaxPlayersProvided ? {
Name = "MAX_PLAYERS"
Value = "${var.max_players}"
} : null,
local.ViewDistanceProvided ? {
Name = "VIEW_DISTANCE"
Value = "${var.view_distance}"
} : null,
local.GameModeProvided ? {
Name = "MODE"
Value = "${var.game_mode}"
} : null,
local.LevelTypeProvided ? {
Name = "LEVEL_TYPE"
Value = "${var.level_type}"
} : null,
local.EnableRollingLogsProvided ? {
Name = "ENABLE_ROLLING_LOGS"
Value = "${var.enable_rolling_logs}"
} : null,
local.TimezoneProvided ? {
Name = "TZ"
Value = "${var.timezone}"
} : null
]
}
]
}
resource "aws_scheduler_schedule_group" "cloud_watch_log_group" {
count = locals.LogGroupNameProvided ? 1 : 0
name = "${var.log_group_name}"
// CF Property(RetentionInDays) = "${var.log_group_retention_in_days}"
}
resource "aws_iam_role" "set_dns_record_lambda_role" {
count = locals.DnsConfigEnabled ? 1 : 0
assume_role_policy = {
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = {
Service = [
"lambda.amazonaws.com"
]
}
Action = [
"sts:AssumeRole"
]
}
]
}
managed_policy_arns = [
"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
]
force_detach_policies = [
{
PolicyName = "root"
PolicyDocument = {
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = "route53:ChangeResourceRecordSets"
Resource = "arn:aws:route53:::hostedzone/${var.hosted_zone_id}"
},
{
Effect = "Allow"
Action = "ec2:DescribeInstance*"
Resource = "*"
}
]
}
}
]
}
resource "aws_lambda_function" "set_dns_record_lambda" {
count = locals.DnsConfigEnabled ? 1 : 0
environment {
variables = {
HostedZoneId = var.hosted_zone_id
RecordName = var.record_name
}
}
code_signing_config_arn = {
ZipFile = "import boto3
import os
def handler(event, context):
new_instance = boto3.resource('ec2').Instance(event['detail']['EC2InstanceId'])
boto3.client('route53').change_resource_record_sets(
HostedZoneId= os.environ['HostedZoneId'],
ChangeBatch={
'Comment': 'updating',
'Changes': [
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': os.environ['RecordName'],
'Type': 'A',
'TTL': 60,
'ResourceRecords': [
{
'Value': new_instance.public_ip_address
},
]
}
},
]
})
"
}
description = "Sets Route 53 DNS Record for Minecraft"
function_name = "${local.stack_name}-set-dns"
handler = "index.handler"
memory_size = 128
role = aws_iam_role.set_dns_record_lambda_role.arn
runtime = "python3.7"
timeout = 20
}
resource "aws_vpc_security_group_ingress_rule" "launch_event" {
count = locals.DnsConfigEnabled ? 1 : 0
// CF Property(EventPattern) = {
// source = [
// "aws.autoscaling"
// ]
// detail-type = [
// "EC2 Instance Launch Successful"
// ]
// detail = {
// AutoScalingGroupName = [
// "${local.stack_name}-asg"
// ]
// }
// }
// CF Property(Name) = "${local.stack_name}-instance-launch"
// CF Property(State) = "ENABLED"
// CF Property(Targets) = [
// {
// Arn = aws_lambda_function.set_dns_record_lambda.arn
// Id = "${local.stack_name}-set-dns"
// }
// ]
}
resource "aws_lambda_permission" "launch_event_lambda_permission" {
count = locals.DnsConfigEnabled ? 1 : 0
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.set_dns_record_lambda.arn
principal = "events.amazonaws.com"
source_arn = aws_vpc_security_group_ingress_rule.launch_event.arn
}
output "check_instance_ip" {
description = "To find your Minecraft instance IP address, visit the following link. Click on the instance to find its Public IP address."
value = "https://${data.aws_region.current.name}.console.aws.amazon.com/ec2/v2/home?region=${data.aws_region.current.name}#Instances:tag:aws:autoscaling:groupName=${aws_neptune_subnet_group.auto_scaling_group.id};sort=tag:Name"
}Still needs lots of work! But hopefully it saves you some time =)
shadycuz
commented
1 year ago I think there is nothing left to do here, closing.
HI there,
I'm currently getting the following error when trying to convert a Cloudformation file to Terraform
The Cloudformation file I used can bee seen here: https://github.com/vatertime/minecraft-spot-pricing/blob/master/cf.yml. I have had to edit the subnets so that cf2tf can parse it (could be a rookie error). Tested on both Windows 11 running Python 3.9.12 and Ubuntu 20.04 running Python 3.8.10