search

hazelcast / hazelcast-aws

AWS EC2 discovery plugin for hazelcast
Other
38 stars 50 forks source link

Fargate support #86

Closed nordri closed 4 years ago

nordri commented 5 years ago

Hi,

I'm trying to deploy Hazelcast cluster on AWS Fargate but I couldn't find any documentation about the process... Does Hazelcast suport Fargate or it only supports ECS?

All the best!

mesutcelik commented 5 years ago

Hi @nordri,

Thanks for the heads up!

Hazelcast only supports for ECS for the time being. However, we see an interest in Fargate too so we definitely consider supporting it.

nordri commented 5 years ago

Thanks for the quick response! OK!

mesutcelik commented 5 years ago

Lets keep it open to make it more visible.

masterjg commented 5 years ago

We also need Fargate support. Cyclos is using hazelcast to support multiple nodes. Right now we are forced to use only single node of Cyclos in production :(

jorgeuriarte commented 5 years ago

Yup, easy cluster discovery in Fargate would be a killer... is there any active work on this? Any workaround we could try/check?

PabloLosky commented 5 years ago

My project is using vertx nodes on a Fargate cluster... is there other options I can use to cluster this verticles ?

alucarpd86 commented 5 years ago

i'm also interested in hazelcast support for Fargate! any news?

mesutcelik commented 5 years ago

Hi,

I have tried to deploy Hazelcast with Container (Local) Networking mentioned in this blog post and it works as expected. However, it is not recommended to use hazelcast in such topology.

What you have to do is to :

  1. create fargate cluster
  2. create fargate task with multiple containers in it like hazelcast1,hazelcast2 using hazelcast base docker image on DockerHub
  3. create a service with the task created but only choose 1 as number of tasks
  4. check CloudWatch logs to see 2 hazelcast members cluster.

Pros

Cons

The other option mentioned in the blog post is External Networking or Task-to-Task communication. This is recommended use case for hazelcast deployments because it offers elasticity features like scale-in/scale-out via manual or auto scaling.However, this requires DescribeTasks API implementation in the Hazelcast AWS Plugin. This will hopefully be implemented in coming months but any community contribution is highly appreciated!

darkredz commented 5 years ago

interested in hazelcast support for Fargate! Hopefully something will be up

googlielmo commented 5 years ago

@darkredz I actually started implementation of Fargate support, which is based on External Networking. I'll open a PR as soon as it's ready for review. HTH :-)

masterjg commented 5 years ago

@darkredz I actually started implementation of Fargate support, which is based on External Networking. I'll open a PR as soon as it's ready for review. HTH :-)

Any ETA? :)

googlielmo commented 5 years ago

@masterjg We'll probably have a first PR by next week, so I'd say 2-3 weeks :)

ls-urs-keller commented 5 years ago

We are using a custom discovery strategy: https://github.com/iKentoo/hazelcast-aws-ecs You can give it a spin using jitpack: https://jitpack.io/#iKentoo/hazelcast-aws-ecs

asakchris commented 5 years ago

We are using client-server topology. Hazalcast server cluster members are setup in EC2 instances and AwsDiscoveryStrategy is used to identify cluster members. Some Hazelcast client runs in EC2 instances and it uses AwsDiscoveryStrategy to idenitfy Hazalcast server cluster members and it works fine. Now I am trying to setup an application in ECS Fargate cluster and used AwsDiscoveryStrategy to identify Hazalcast server cluster members but it is failed. Please let me know whether fix in this PR will resolve this issue or not. Thank you in advance.

googlielmo commented 5 years ago

Hi @asakchris The upcoming ECS support PR is designed to address the discovery of Hazelcast member instances running on ECS. For client-server topology, the clients can be located either on ECS or outside of it (provided the appropriate networking security policies are in place). That said, I don't see any problem for client applications running on ECS to discover servers running on EC2 instances (the opposite setup to what I described above). Hope this helps. Cheers, Guglielmo

asakchris commented 5 years ago

Hi @googlielmo Below is the configuration used in client running in ECS to discover Hazelcast server cluster members. It is not working.

    <network>
        <connection-attempt-period>60000</connection-attempt-period>
        <connection-attempt-limit>2147483647</connection-attempt-limit>
        <discovery-strategies>
            <discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">
                <properties>
                    <property name="region">us-east-1</property>
                    <property name="tag-key">ApplicationType</property>
                    <property name="tag-value">DEV-HAZELCAST-CACHE-CLUSTER</property>
                    <property name="hz-port">5701</property>
                </properties>
            </discovery-strategy>
        </discovery-strategies>
    </network>

Find below logs:

2019-06-12 20:31:25,481 WARN [de-web-client.cluster-] com.hazelcast.aws.utility.RetryUtils [ ]: Couldn't connect to the AWS service, [10] retrying in 57 seconds...
2019-06-12 20:32:23,140 WARN [de-web-client.cluster-] c.hazelcast.aws.AwsDiscoveryStrategy [ ]: Cannot discover nodes, returning empty list
java.net.ConnectException: Invalid argument (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:70)
... 23 common frames omitted
Wrapped by: com.hazelcast.config.InvalidConfigurationException: Unable to lookup role in URI: http://169.254.169.254/latest/meta-data/iam/security-credentials/
at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:78)
at com.hazelcast.aws.utility.MetadataUtil$1.call(MetadataUtil.java:109)
at com.hazelcast.aws.utility.MetadataUtil$1.call(MetadataUtil.java:106)
at com.hazelcast.aws.utility.RetryUtils.retry(RetryUtils.java:52)
at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:106)
at com.hazelcast.aws.impl.DescribeInstances.retrieveRoleFromURI(DescribeInstances.java:170)
at com.hazelcast.aws.impl.DescribeInstances.getDefaultIamRole(DescribeInstances.java:124)
at com.hazelcast.aws.impl.DescribeInstances.fillKeysFromIamRoles(DescribeInstances.java:109)
at com.hazelcast.aws.impl.DescribeInstances.execute(DescribeInstances.java:254)
at com.hazelcast.aws.AWSClient.getAddresses(AWSClient.java:57)
at com.hazelcast.aws.AwsDiscoveryStrategy.discoverNodes(AwsDiscoveryStrategy.java:146)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:74)
at com.hazelcast.client.spi.impl.discovery.DiscoveryAddressProvider.loadAddresses(DiscoveryAddressProvider.java:44)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.getPossibleMemberAddresses(ClientConnectionManagerImpl.java:901)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.connectToClusterInternal(ClientConnectionManagerImpl.java:817)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl.access$1200(ClientConnectionManagerImpl.java:104)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$2.call(ClientConnectionManagerImpl.java:864)
at com.hazelcast.client.connection.nio.ClientConnectionManagerImpl$2.call(ClientConnectionManagerImpl.java:860)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
2019-06-12 20:32:23,141 WARN [de-web-client.cluster-] c.h.c.c.ClientConnectionManager [ ]: de-web-client [hazelcast-dev] [3.9.4] Unable to get alive cluster connection, try in 0 ms later, attempt 1 of 2147483647.

It appears that the meta-data HTTP endpoint is not available to containers running in Fargate, refer here . So AwsDiscoveryStrategy is not working. Please let me know if you need more details.

googlielmo commented 5 years ago

Hi @asakchris

Are you talking about the unmerged PR #95?

I ask because this issue is all about adding ECS support to the current AWS plugin. At the moment ECS is not supported outside of some simple scenarios.

We are aware of the different metadata APIs on ECS / Fargate and are going to implement a different discovery strategy in the PR above.

See also @mesutcelik's comment above (https://github.com/hazelcast/hazelcast-aws/issues/86#issuecomment-453255537)

Goinidias commented 5 years ago

Far enough along to have a sense of when when PR #95 will be ready?

googlielmo commented 5 years ago

Hi @Goinidias! We don't really know at the moment to be honest, we were busy with some other things and we're still collecting and considering requirements and suggestions from the community (anyone with comments or suggestions etc. please step forward). "Shouldn't take long" has become a cliché in software development, so I'll avoid it :-)

Cheers, Guglielmo

asakchris commented 5 years ago

Hi @googlielmo

I didn't find description of PR #95.

Hazelcast client applications running on ECS Fargate is not able to discover Hazelcast servers running on EC2 instances using tags. See (#86 (comment)) for configuration and exception trace.

Below is the task role of Hazelcast client running in ECS Fargate:

  ECSTaskRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: ECS-TASK-ROLE
      AssumeRolePolicyDocument:
        Statement:
        - Effect: 'Allow'
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Policies:
      - PolicyName: ECSServiceDiscovery
        PolicyDocument:
          Statement:
          - Effect: Allow
            Action:
              # Allow ECS container to access ECS tasks
              - 'ecs:ListTasks'
              - 'ecs:DescribeTasks'
              # Allow ECS tasks to get EC2 instances tag information to identify hazelcast server
              - 'ec2:DescribeInstances'
            Resource: '*'
googlielmo commented 5 years ago

Hi @asakchris indeed it looks like a different issue (this one is more about adding support for Hazelcast clusters running on ECS, not clients). It would be good to open a separate issue so that we can track it separately. Also, please specify exact version of Hazelcast and AWS plugin. Thanks!

asakchris commented 4 years ago

Thank you @googlielmo I will open new issue.