hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.82k stars 9.17k forks source link

Get Private IP of instance launched by autoscaling group #511

Open hashibot opened 7 years ago

hashibot commented 7 years ago

This issue was originally opened by @coolgooze as hashicorp/terraform#11713. It was migrated here as part of the provider split. The original body of the issue is below.


ASG resource "aws_autoscaling_group" "REDIS_ASG" { name = "${var.environment}-REDIS_ASG" launch_configuration = "${aws_launch_configuration.redis_launch_conf.name}"

availability_zones = ["us-west-2a", "us-west-2b", "us-west-2c"]

vpc_zone_identifier = ["${data.aws_subnet.PrivateDBSubnetAZ1.id}","${data.aws_subnet.PrivateDBSubnetAZ2.id}","${data.aws_subnet.PrivateDBSubnetAZ3.id}"]
min_size = 1
max_size = 1
desired_capacity = 1
health_check_grace_period = 600
health_check_type = "EC2"   
force_delete = "false"
termination_policies = ["OldestInstance"]
tag {
key                 = "Name"
value               = "${var.environment}-int-redis"
propagate_at_launch = true
 }

}

i am trying to create a A Record resource "aws_route53_record" "redis" { zone_id = "${data.aws_route53_zone.dns.zone_id}" name = "redis-${var.environment}.${data.aws_route53_zone.dns.name}" type = "A" ttl = "60" records = ["${aws_autoscaling_group.REDIS_ASG.private_ip}"] }

Output-

jasonkuehl commented 7 years ago

I just had a need for this today. I would love for this to be a real thing.

brandonawells commented 7 years ago

I ran into a need for this today. I'd love this to exist. Any updates on the request?

catsby commented 7 years ago

Hello – Can you describe the use case here? I'm curious why you would want the direct IP of any instance that's in an autoscaling group, as opposed to connecting via a Load Balancer.

If I'm not mistaken a specific advantage of an ASG is ensuring you have a certain number of instances available and running, not that any given one of them exists at a certain IP.

If the ASG resource were to export a private IP, and on AWS side the ASG were to change instances for any reason, that IP could become stale until your next plan and apply, right? In which case there could be a window of time when your A record routes you to a bad IP, correct? Where as if the A record pointed to an IP that's assigned to a load balance, the roll over would happen automatically.

I'm curious how exposing this would be used in a reliable, stable way. Thanks!

jasonkuehl commented 7 years ago

@catsby You are correct in saying that IP would be stale and Terraform would need to run in order to update the A record attached to the Route53 record.

We have use case that we have a set of servers that run different jobs. Everything is controlled in puppet and Terraform. Right now we have a module that builds off a count (normally). What we wanted to do was put them into an ASG in order to have self healing in case a node would fail. However we cant move forward with testing due to there is no output of private ips or a node list.

We started an effect sometime ago to remove IP Address from configs and swap to DNS names. It would be icing on the cake if we can build a way to self heal these servers in case of aws instance failure using ASG.

jainishshah17 commented 7 years ago

+1

bdirito commented 7 years ago

Another use case https://github.com/terraform-community-modules/tf_aws_bastion_s3_keys

This sets up a bastion in an autoscaling group min=max=desired=1. The reasoning behind this is to autorelaunch the bastion if it goes down. W/o this however I have to do a check after to see what ip the bastion actually got.

bernadinm commented 7 years ago

It would be very useful to have this feature. Users can run types of one off workloads with a list of known IPs for the ASG within a null_resource. +1 for me as well.

igoratencompass commented 6 years ago

@catsby Look at my comment in the closed issue migrated here: https://github.com/hashicorp/terraform/issues/11713

To repeat:

Very useful, will provide same functionality as CloudFormation does. We can simply wait for ASG to finish creating all instances and report back with private ip's, instance id's etc. so we can have further actions follow in terraform like creating dns records for the instances etc.

Also no one says that ASG has to be related to an ELB, in our case we have the instances behind HAProxy so we need to know their IP's.

This has been opened since February, is it really that difficult to resolve?

igoratencompass commented 6 years ago

@catsby Another example, I want to create Route53 DNS records and health checks after launching instances:

# Host-specific A records for both load-balancers
resource "aws_route53_record" "service-record" {
   zone_id = "<my-zone-id>"
   count = 2
   name = "service-lb-${format("%03d", count.index + 1)}.mydomain.com"
   type = "A"
   ttl = "60"
   records = ["${element(aws_instance.lb-service.*.public_ip, count.index)}"]
}

# Health checks for each of the load balancers
resource "aws_route53_health_check" "service-healthcheck" {
  ip_address = "${element(aws_instance.lb-service.*.public_ip, count.index)}"
  count = 2
  port = 50000
  type = "HTTP"
  resource_path = "/health"
  failure_threshold = "5"
  request_interval = "30"
  tags = {
    Name = "service-${format("%03d", count.index + 1)}.production"
  }
}

# Group consisting of 2 alias records to the host records, with associated health checks, having weighted routing
resource "aws_route53_record" "service-group" {
   zone_id = "<my-zone-id>"
   count = 2
   name = "service.mydomain.com"
   type = "A"
   weighted_routing_policy = {
      weight = "50"
   }
   health_check_id = "${element(aws_route53_health_check.service-healthcheck.*.id, count.index)}"
   set_identifier = "service${format("%03d", count.index + 1)}"
   alias {
     name = "${element(aws_route53_record.service-record.*.fqdn, count.index)}"
     zone_id = "<my-zone-id>"
     evaluate_target_health = true
   }
}

When instances are launched via ASG there is no way to do this. We are forced to move this code into user-data instead which is big PITA.

hernan82arg commented 6 years ago

+1

igoratencompass commented 6 years ago

Any update on this?

catsby commented 6 years ago

I'm not opposed to this feature in general but I do have opinions on how it should be implemented. I'll also be upfront and say that we (HashiCorp) are unlikely to get to this feature soon, but gladly welcome contributions if someone can pick it up.

As for how to implement:

So the problem here in short is that the IP Addresses are not returned by default and you need to do a handful of additional API calls to get them. For large setups, these extra API calls add up quickly, so by default, we should not include Instance IP Addresses in the state of AutoScaling Groups. I recognize it's a useful feature but also think it will not be an often used one, so by default we shouldn't consume all these API calls.

That said, I believe/propose we can support this functionality with a data source:

Alternatively we could expand the data.aws_autoscaling_groups data source, but I think I'd prefer the separation.

To get the IP address(es) for the Instance(s) we need to call DescribeInstances for each one. Fortunately we can use an EC2Filter and filter by requester-id, in this case, the AutoScaling Group Id, so we don't have to group up the instance-ids from the DescribeASG output and then feed that into the DescribeInstances call.

I believe this new data source would keep this instance information in a TypeSet of Instances. Unfortunately you can't directly reference the value in a set with things like element(set, count). So if you want a list of ip addresses, we may want to take the results of the DescribeInstances call and aggregate all the public ips into a calculated field public_ips , and private ips into private_ips.

So in the end the out put of this data source would be like this:

How does this sound?


resource "autoscaling_group" "example" {
  [...]
}

data "autoscaling_group" "example_ips" {
  autoscaling_id = "${aws_autoscaling_group.example.id}"
  # Fetch Instance information
  # Retrieves instance information
  # Retrieves public_ip's
  # Retrieves private_ip's
  get_instance_properties = true
}

resource "aws_route53_record" "service-record" {
   zone_id = "<my-zone-id>"
   count = 2
   name = "service-lb-${format("%03d", count.index + 1)}.mydomain.com"
   type = "A"
   ttl = "60"
   records = ["${element(data.aws_autoscaling_group.example_ips.private_ips, count.index)}"]
}
jasonkuehl commented 6 years ago

@catsby I love this and it would help a lot.

igoratencompass commented 6 years ago

@catsby beautiful!

samsixtyone commented 6 years ago

any update on this, in real need of this solution.

im2kl commented 6 years ago

Has this been looked at? a complete stopper on what im required to do.

alona-shevliakova commented 6 years ago

any update on this?

peterloron commented 6 years ago

This would be amazing to have!

n3ph commented 6 years ago

This would be glorious! Since Lambda triggered by CloudWatchEvent is a mess!

lebedevdsl commented 6 years ago

Need this one too

SushantJagdale commented 6 years ago

Anyone find workaround or solution for this issue? This is completely show stopper for what I need to do.

JeremieBethmont commented 6 years ago

+1

xvillanu commented 6 years ago

+1

uriinf commented 6 years ago

+1

drnorton83 commented 6 years ago

+1

cleanshavenalex commented 6 years ago

+1

Alifener commented 6 years ago

+1

tomaszwostal commented 6 years ago

+1

sadok-f commented 6 years ago

+1

tlipatov commented 6 years ago

Here are some solutions that I implement:

  1. Use the module terraform-aws-modules/autoscaling/aws to launch my ASG's Tag the ASG and then use the aws_instance data source to search for the instances based on the tags. Then do what ever I need to do with the data

  2. When using an ASG for a bastion host. I create an iam_instance_profile for the ASG that allows the instance itself to update its own route53 record when it boots up using a launch config script.

Hope this helps

JoaoManoel commented 6 years ago

+1

ricardo-larosa commented 6 years ago

+1

lexsys27 commented 6 years ago

Is there currently any workaround available?

lexsys27 commented 6 years ago

This way allowed me to get ips of worker nodes:

data "aws_instances" "workers" {
  instance_tags {
    Name = "lexsys-eks-asg"
  }
}

output "private-ips" {
  value = "${data.aws_instances.workers.private_ips}"
}

output "public-ips" {
  value = "${data.aws_instances.workers.public_ips}"
}
Outputs:

private-ips = [
    10.0.0.75,
    10.0.1.87
]
public-ips = [
    52.44.215.211,
    54.153.70.110
]

Sure, this workaround has the limitations, but worked for me.

ghost commented 6 years ago

@catsby One use case is for cluster setup. In such a scenario we want to be able to only provide the SSH keys of the instances to people who want to test a cluster like kafka, zookeeper, elasticsearch etc without having to give them access to the AWS console. If terraform could simply output the private ip address then the troubleshooters would have a list of IP addresses to log into simply with their SSH keys and without requiring access to AWS Console.

nunofernandes commented 6 years ago

+1

moosahmed commented 6 years ago

@catsby Yeah another use case is when making a kubernetes cluster through an asg. you need the master's private ip to feed into the workers. Is there someone who knows a work around for this?

matt9949 commented 6 years ago

+1

npockrus-bnet commented 6 years ago

+1

jeandreh commented 6 years ago

+1. My use case involves Hashicorp Vault, which needs to be initialised before being recognised as a healthy node by the load balancer.

kamusin commented 6 years ago

+1

KIVagant commented 6 years ago

Any workarounds for Google Cloud Platform? It has the same problem with google_compute_instance_group_manager.

igoratencompass commented 6 years ago

@lexsys27 the question is specific for when instances are created via aws_autoscaling_group

@KIVagant I'm afraid you'll need to wait patiently as the rest of us have (since Feb 2017) for this to be included in terraform, my prognoses would be around version 2.27 ... probably :-)

lexsys27 commented 6 years ago

@igoratencompass I do use aws_autoscaling_group to create instances.

I tag the instances in the template and use this tag to retrieve IPs in the data section.

igoratencompass commented 6 years ago

I see was not aware of that feature of data module.

RuBiCK commented 5 years ago

https://github.com/terraform-providers/terraform-provider-aws/issues/511#issuecomment-401362499 is a workaround but be carefull doc page

Note: It's strongly discouraged to use this data source for querying ephemeral instances (e.g. managed via autoscaling group), as the output may change at any time and you'd need to re-run apply every time an instance comes up or dies.

This note is other reason to implement this new data source

syncroswitch commented 5 years ago

you can pull all instance data with a round about lookup.

( example assuming some asg named aws_autoscaling_group.one is already defined )

data "aws_instances" "nodes" {
  depends_on = [ "aws_autoscaling_group.one" ]

  instance_tags {
    Name = "some_unique_tag_in_your_asg"
  }
}

data "aws_instance" "asg-one-instances" {
  count = "${ var.asg_one_count }"
  depends_on = ["data.aws_instances.nodes"]
  instance_id = "${data.aws_instances.nodes.ids[count.index]}"
}

output "private-ips" {
  value = "${ data.aws_instance.asg-one-instances.*.private_ip }"
}

output "public-ips" {
  value = "${ data.aws_instance.asg-one-instances.*.public_ip }"
}

output "private-dnsnames" {
  value = "${ data.aws_instance.asg-one-instances.*.private_dns }"
}

output "public-dnsnames" {
  value = "${ data.aws_instance.asg-one-instances.*.public_dns }"
}

but I would much prefer to have the instance data directly available in the asg attributes.

igoratencompass commented 5 years ago

@syncroswitch the problem I see with this is the asg_one_count variable. It is a static value which will/may not match the actual state of the ASG on consecutive runs, i.e. teh ASG has grown or shrunk in the mean time.

claywd commented 5 years ago

@syncroswitch @igoratencompass it works to get the endpoints and private ips when you launch, which is what is mostly needed.

Also, just set count = "${ var.asg_one_count }" to count = ${aws_autoscaling_group.one.desired_capacity}

If people truly have need of dynamic ips then they should write it up in python, bash, go, etc and call it from terraform then use that instead of the desired capacity but I doubt it will really be necessary or desired in the long run.

minhajuddin commented 4 years ago

I started off with @syncroswitch's config and ended with a simpler workaround:

data "aws_instances" "ecs_instances_meta" {
  instance_tags = {
    # Use whatever name you have given to your instances
    Name = var.ecs_cluster_name
  }
}

output "ecs-private-ips" {
  value = data.aws_instances.ecs_instances_meta.private_ips
}