AWS ECS takes 5 minutes(300 seconds) to find members

Finfox2019 commented 5 years ago

I am running a Vert.x with Hazelcast 3.10.5 (default) on AWS ECS using hazelcast-aws plugin 2.2. For old hazelcast version (3.8) and plugin 2.1.0, it worked fine, but with new hazelcast version, it took 5 minutes (300 seconds) to find other members. Locally, it can immediately join other members.

Here is configuration xml file

<aws enabled="false">
</aws>
<discovery-strategies>
<discovery-strategy
    enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
  <properties>
    <property name="iam-role">my-iam-role</property>
    <property name="region">us-west-2</property>
    <property name="host-header">ec2.amazonaws.com</property>
    <property name="tag-key">myCluster</property>
    <property name="tag-value">myValue</property>
  </properties>
</discovery-strategy>
</discovery-strategies>
</join>
<interfaces enabled="true">
<interface>10.10.*.*</interface>
</interfaces>

Here is log

11:30:00 2019-08-01T18:30:00,090 [INFO ] LifecycleService: - [10.10.2.41]:5701 [dev] [3.10] [10.10.2.41]:5701 is STARTING
11:30:00 2019-08-01T18:30:00,161 [INFO ] system: - [10.10.2.41]:5701 [dev] [3.10] Cluster version set to 3.10
11:30:00 2019-08-01T18:30:00,162 [INFO ] ClusterService: - [10.10.2.41]:5701 [dev] [3.10]
Members {size:1, ver:1} [
Member [10.10.2.41]:5701 - 1be4ca1e-4792-4d3a-a960-e73ad073e2a6 this]

11:35:00 2019-08-01T18:35:00,442 [INFO ] TcpIpConnectionManager: - [10.10.2.41]:5701 [dev] [3.10] Established socket connection between /10.10.2.41:37510 and /10.10.2.41:5702

Members {size:2, ver:2} [
Member [10.10.2.41]:5701 - 1be4ca1e-4792-4d3a-a960-e73ad073e2a6 this
Member [10.10.2.41]:5702 - 813f9154-e8e3-4c82-83b4-6921ec80519b]

As you can see, 5 minutes later, 11:35 it established connection.

SushmaReddyLoka commented 4 years ago

Hi , I faced similar issue. When i used tag key and value instead of security group discovery is taking more time. So is it expected that discovery takes time without security group setting and just using tag key and value ?

alparslanavci commented 4 years ago

Hi @Finfox2019 and @SushmaReddyLoka, do you have any reproducers for the issue? If so, can you please share it?

neilstevenson commented 4 years ago

hazelcast / hazelcast-aws

AWS ECS takes 5 minutes(300 seconds) to find members #104