search

Codemill / av-ecs-cloudformation

Cloudformation templates for deploying Accurate Video to AWS
1 stars 0 forks source link

aws hazelcast discovery not working #3

Closed peter-ahlstrom closed 3 years ago

peter-ahlstrom commented 4 years ago 
 __  ____  __  _____   ___  __ ____  ______                                                                                                                                                                                                                                                   
  --/ __ \/ / / / _ | / _ \/ //_/ / / / __/                                                                                                                                                                                                                                                   
  -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \                                                                                                                                                                                                                                                     
 --\___\_\____/_/ |_/_/|_/_/|_|\____/___/                                                                                                                                                                                                                                                     
 2020-08-11 07:58:26,948 INFO  [org.fly.cor.int.lic.VersionPrinter] (main) Flyway Community Edition 6.4.2 by Redgate                                                                                                                                                                          
 2020-08-11 07:58:30,843 INFO  [org.fly.cor.int.dat.DatabaseFactory] (main) Database: jdbc:postgresql://ad7hlun7sopy1x.ctsayudz0tc6.eu-north-1.rds.amazonaws.com/accurateVideo (PostgreSQL 11.6)                                                                                              
 2020-08-11 07:58:31,543 INFO  [org.fly.cor.int.com.DbMigrate] (main) Current version of schema "public": 1.0.14                                                                                                                                                                              
 2020-08-11 07:58:31,545 INFO  [org.fly.cor.int.com.DbMigrate] (main) Schema "public" is up to date. No migration necessary.                                                                                                                                                                  
 2020-08-11 07:58:37,945 WARNING [com.haz.con.AbstractXmlConfigHelper] (vert.x-worker-thread-0) Name of the hazelcast schema location is incorrect, using default                                                                                                                             
 2020-08-11 07:58:40,736 INFO  [com.haz.ins.AddressPicker] (vert.x-worker-thread-0) [LOCAL] [dev] [3.12.7] Interfaces is enabled, trying to pick one address matching to one of: [10.0.*.*]                                                                                                   
 2020-08-11 07:58:40,736 INFO  [com.haz.ins.AddressPicker] (vert.x-worker-thread-0) [LOCAL] [dev] [3.12.7] Prefer IPv4 stack is true, prefer IPv6 addresses is false                                                                                                                          
 2020-08-11 07:58:40,741 INFO  [com.haz.ins.AddressPicker] (vert.x-worker-thread-0) [LOCAL] [dev] [3.12.7] Picked [10.0.2.101]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true                                                                          
 2020-08-11 07:58:40,847 INFO  [com.haz.system] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] Hazelcast 3.12.7 (20200430 - bf4b643) starting at [10.0.2.101]:5701                                                                                                                 
 2020-08-11 07:58:40,847 INFO  [com.haz.system] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.                                                                                                                       
 2020-08-11 07:58:42,345 INFO  [com.haz.spi.imp.ope.imp.BackpressureRegulator] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] Backpressure is disabled                                                                                                                             
 2020-08-11 07:58:42,555 WARNING [com.haz.int.con.ConfigValidator] (vert.x-worker-thread-0) Property hazelcast.rest.enabled is deprecated. Use configuration object/element instead.                                                                                                          
 2020-08-11 07:58:42,555 WARNING [com.haz.int.con.ConfigValidator] (vert.x-worker-thread-0) Property hazelcast.memcache.enabled is deprecated. Use configuration object/element instead.                                                                                                      
 2020-08-11 07:58:44,239 WARNING [com.haz.aws.AwsDiscoveryStrategy] (vert.x-worker-thread-0) Describe instances will be queried with iam-role assigned to EC2 instance, please make sure given iam-role have ec2:DescribeInstances policy attached.                                           
 2020-08-11 07:58:44,643 INFO  [com.haz.ins.Node] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] Activating Discovery SPI Joiner                                                                                                                                                   
 2020-08-11 07:58:46,150 INFO  [com.haz.spi.imp.ope.imp.OperationExecutorImpl] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)                                                                  
 2020-08-11 07:58:46,153 INFO  [com.haz.int.dia.Diagnostics] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.                                                                         
 2020-08-11 07:58:46,239 INFO  [com.haz.cor.LifecycleService] (vert.x-worker-thread-0) [10.0.2.101]:5701 [dev] [3.12.7] [10.0.2.101]:5701 is STARTING                                                                                                                                         
 2020-08-11 07:58:46,341 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [1] retrying in 1 seconds...                                                                                                                                      
 2020-08-11 07:58:47,842 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [2] retrying in 2 seconds...                                                                                                                                      
 2020-08-11 07:58:50,093 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [3] retrying in 3 seconds...                                                                                                                                      
 2020-08-11 07:58:53,468 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [4] retrying in 5 seconds...                                                                                                                                      
 2020-08-11 07:58:58,531 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [5] retrying in 7 seconds...                                                                                                                                      
 2020-08-11 07:59:06,125 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [6] retrying in 11 seconds...                                                                                                                                     
 2020-08-11 07:59:17,515 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [7] retrying in 17 seconds...                                                                                                                                     
 2020-08-11 07:59:34,599 WARNING [com.haz.aws.uti.RetryUtils] (vert.x-worker-thread-0) Couldn't connect to the AWS service, [8] retrying in 25 seconds...                                                                                                                                     
 2020-08-11 07:59:38,939 WARNING [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-0,5,main]=Thread[vert.x-worker-thread-0,5,main] has been blocked for 61495 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked  
  at java.base@14.0.2/java.lang.Thread.sleep(Native Method)                                                                                                                                                                                                                                   
  at app//com.hazelcast.aws.utility.RetryUtils.sleep(RetryUtils.java:79)                                                                                                                                                                                                                      
  at app//com.hazelcast.aws.utility.RetryUtils.retry(RetryUtils.java:61)                                                                                                                                                                                                                      
  at app//com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:106)                                                                                                                                                                                               
  at app//com.hazelcast.aws.AWSClient.getAvailabilityZone(AWSClient.java:62)                                                                                                                                                                                                                  
  at app//com.hazelcast.aws.AwsDiscoveryStrategy.discoverLocalMetadata(AwsDiscoveryStrategy.java:138)                                                                                                                                                                                         
  at app//com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverLocalMetadata(DefaultDiscoveryService.java:88)                                                                                                                                                                     
  at app//com.hazelcast.instance.Node.mergeEnvironmentProvidedMemberMetadata(Node.java:589)                                                                                                                                                                                                   
  at app//com.hazelcast.instance.Node.start(Node.java:434)                                                                                                                                                                                                                                    
  at app//com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136)                                                                                                                                                                                                 
  at app//com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)                                                                                                                                                                       
  at app//com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)                                                                                                                                                                             
  at app//com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)                                                                                                                                                                             
  at app//com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)                                                                                                                                                                                                                
  at app//io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.lambda$join$1(HazelcastClusterManager.java:136)                                                                                                                                                                              
  at app//io.vertx.spi.cluster.hazelcast.HazelcastClusterManager$$Lambda$353/0x00000008010b8c40.handle(Unknown Source)                                                                                                                                                                        
  at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316)                                                                                                                                                                                                       
  at app//io.vertx.core.impl.ContextImpl$$Lambda$354/0x0000000801150440.run(Unknown Source)                                                                                                                                                                                                   
  at app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)                                                                                                                                                                                                                                 
  at app//io.vertx.core.impl.TaskQueue$$Lambda$351/0x00000008010b9440.run(Unknown Source)                                                                                                                                                                                                     
  at java.base@14.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)                                                                                                                                                                                         
  at java.base@14.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)                                                                                                                                                                                         
  at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                                                                                                                                                                               
  at java.base@14.0.2/java.lang.Thread.run(Thread.java:832)                                                                                                                                                                                                                                   
 2020-08-11 07:59:40,939 WARNING [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-0,5,main]=Thread[vert.x-worker-thread-0,5,main] has been blocked for 63495 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked  
  at java.base@14.0.2/java.lang.Thread.sleep(Native Method)                                                                                                                                                                                                                                   
  at app//com.hazelcast.aws.utility.RetryUtils.sleep(RetryUtils.java:79)                                                                                                                                                                                                                      
  at app//com.hazelcast.aws.utility.RetryUtils.retry(RetryUtils.java:61)                                                                                                                                                                                                                      
  at app//com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:106)                                                                                                                                                                                               
  at app//com.hazelcast.aws.AWSClient.getAvailabilityZone(AWSClient.java:62)                                                                                                                                                                                                                  
  at app//com.hazelcast.aws.AwsDiscoveryStrategy.discoverLocalMetadata(AwsDiscoveryStrategy.java:138)                                                                                                                                                                                         
  at app//com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverLocalMetadata(DefaultDiscoveryService.java:88)                                                                                                                                                                     
  at app//com.hazelcast.instance.Node.mergeEnvironmentProvidedMemberMetadata(Node.java:589)                                                                                                                                                                                                   
  at app//com.hazelcast.instance.Node.start(Node.java:434)                                                                                                                                                                                                                                    
  at app//com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136)                                                                                                                                                                                                 
  at app//com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)                                                                                                                                                                       
  at app//com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)                                                                                                                                                                             
  at app//com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)                                                                                                                                                                             
  at app//com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)                                                                                                                                                                                                                
  at app//io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.lambda$join$1(HazelcastClusterManager.java:136)                                                                                                                                                                              
  at app//io.vertx.spi.cluster.hazelcast.HazelcastClusterManager$$Lambda$353/0x00000008010b8c40.handle(Unknown Source)                                                                                                                                                                        
  at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:316)                                                                                                                                                                                                       
  at app//io.vertx.core.impl.ContextImpl$$Lambda$354/0x0000000801150440.run(Unknown Source)                                                                                                                                                                                                   
  at app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)                                                                                                                                                                                                                                 
  at app//io.vertx.core.impl.TaskQueue$$Lambda$351/0x00000008010b9440.run(Unknown Source)                                                                                                                                                                                                     
  at java.base@14.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)                                                                                                                                                                                         
  at java.base@14.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)                                                                                                                                                                                         
  at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                                                                                                                                                                               
  at java.base@14.0.2/java.lang.Thread.run(Thread.java:832)
peter-ahlstrom commented 4 years ago

Documentation for plugin configuration https://github.com/hazelcast/hazelcast-aws/blob/v2.4/README.md

andreasolund commented 3 years ago

We're working on a fix that is coming soon.

andreasolund commented 3 years ago

Fixed.