moooofly commented 5 years ago

之前通过命令，每 15min 抓取了一次 dig ELB 的输出，从输出上看，ip 列表一直在发生着变化

经过和 aws 技术支持的沟通，了解到通过上述 dig 命令是无法获取到完整 ip 列表的，需要使用如下命令进行 ip 列表获取：

dig neo-app-prod-elb-936082747.cn-north-1.elb.amazonaws.com.cn TXT +short +vc | awk -F "[\"\"]" '{for(i=1;i<=NF;i++) if(length($i)!=1 && length($i)!=0) print $i}' | tr -d '\n' | awk -F "c100e1" '{for(i=1;i<=NF-1;i++) print $i}' | sed s/[[:space:]]//g

其中

neo-app-prod-elb-936082747.cn-north-1.elb.amazonaws.com.cn 是 apineo.llsapp.com 的 CNAME
理论上讲，dig 前者获取到的内容会更准确；

经过改良，可以使用如下命令实现：每一分钟测试一次 elb 全部 ip 地址的连通性

while [ 1 ]; do sleep 60; date +"%T %D %Y"; (dig neo-app-prod-elb-936082747.cn-north-1.elb.amazonaws.com.cn TXT +short +vc | awk -F "[\"\"]" '{for(i=1;i<=NF;i++) if(length($i)!=1 && length($i)!=0) print $i}' | tr -d '\n' | awk -F "c100e1" '{for(i=1;i<=NF-1;i++) print $i}' | sed s/[[:space:]]//g | tee | while read LINE; do nc -vz $LINE 443; done) done

每 15 分钟 dig 一次获取 elb ip 地址列表（前 8 个）

while [ 1 ]; do echo "----"; date +"%T %D %Y"; dig apineo.llsapp.com; sleep 900; done

客户端 app 从 DNSPod HTTPdns 上获取 ip 列表的命令

这个是基于免费接口的调用，真正使用的企业版接口，略微有点差异

curl http://119.29.29.29/d\?dn\=apineo.llsapp.com.

只会拿到 8 个 ip 地址

AWS 的反馈

我在后台查了一下您的这个 ELB 实例，发现因为您这个 ELB 的流量比较大，目前 ELB 的节点已经扩展到18 个了，也就是说 neo-app-prod-elb-936082747.cn-north-1.elb.amazonaws.com.cn 这个域名可能解析出的不同的IP有 18 个之多。

因为 DNS 的策略问题，一般每次 dig 等 dns 解析只会返回 8 个 IP ，在 60s 内的 TTL 结束前，这 8 个是稳定的，但是 60s 后就不一定了，因为会从总共 18 个里挑 8 个返回。所以说，您看到的 ELB 对应的 IP 在非高峰期也在不停发生变化，这个是正常的现象。

其实当 ELB 比较稳定的时候，您访问这 18 个 IP 应该是都可以的，所以问题的关键是为何会有一些客户端无法访问 ELB 。

moooofly commented 5 years ago

ELB is dropping connections?

We appreciate the feedback resulting from your performance tests. We want to provide some information about the Elastic Load Balancing service that should help guide testing and benchmarking efforts:

Clients should leverage a dynamic set of IP addresses that may appear at any time behind the DNS entry for your Load Balancer.
- When creating a Load Balancer, you are given a DNS name for your clients to use to find out where to send traffic. Behind this name, we place a dynamically changing set of IP addresses, of which each of your clients will get one per resolution and use it for a while. Our support for high availability and multiple availability zones, as well as our ability to dynamically scale your load balancing capacity leverages our ability to change the set of IP addresses behind the DNS name from time to time.
- Testing with a single client machine will result in using only one of these IP addresses. When a larger set of independent clients resolve the DNS entry and get different IP addresses to use, they will effectively spread the load amongst the full set.
Your test should ramp up over time and run for long enough to see the benefits of dynamically scaled load balancer capacity. As your traffic increases, load balancing capacity ramps up somewhat smoothly, partially by changing the set of IP addresses behind the DNS entry. These IP addresses may take a bit of time to be picked up and used by your clients. Make sure each of your test clients re-resolve the DNS name periodically as the test progresses.
When using HTTP load balancing, you can maximize performance by enabling persistent connections on your back-end instances. The load balancer then manages the high connection rate from the internet while maintaining a smaller set of stable, persistent connections to your back-end instances. To support this TCP connection offload feature, all you need to do is enable persistent connections in the HTTP server software that you are running on your back-end instances.

In general, I'd suggest that you collect historical data or estimate a traffic profile and how it progresses over a typical (or perhaps extreme, but still real-world) day for your scenario. Then, add enough headroom to the numbers to make you feel comfortable and confident that, if ELB can handle the test gracefully, it can handle your actual workload.

The elements of your traffic profile that you may want to consider in your test strategy may include, for example:

connections / second
concurrent connections
requests / second
concurrent requests
mix of request sizes
mix of response sizes

ELB attempts to stay ahead of an increasing load, so connection attempts should continue to succeed.

You don't need to generate test traffic from an enormous number of clients. But don't generate all your test traffic from one or just a few clients. A dozen clients per configured availability zone should more than do the trick. Just divide your desired load amongst them and make sure each refreshes their DNS resolution results every few minutes.

Having a spread of source IP addresses is not necessary - the main point is that the full set of IP addresses provided by DNS are utilized.

The ELB system strives to scale ahead of demand but wild spikes in traffic demand (such as 0 load to full load swings during initial load testing) can temporarily run past it's provisioned capacity which will result in 503 Service Unavailable until the system can get ahead of demand. Under extreme overload, timeouts can occur. Customer traffic flows are typically gradual swings that occur at rates measured in minutes or hours, not fractions of a second.

moooofly commented 5 years ago

Amazon EC2 Elastic Load Balancer: Does its IP ever Change?

Q: Does the ELB's IP Ever Change once setup, or will it always access instances from the same location during its lifetime no matter what might be going on with it behind the scenes at Amazon?

A: The short answer: Yes, ELB's IP addresses (both the ones that are publicly distributed to clients of your service, and the internal IPs from which ELB sends traffic to your instances) dynamically change. A: The long answer: See my article about how ELB works for more info: http://shlomoswidler.com/2009/07/elastic-in-elastic-load-balancing-elb.html

moooofly commented 5 years ago