hjha287
commented
7 years ago By looking at the error it seems ES is not running on port 9300. Please check ES is running on what port and update the properties file accordingly. is it running on 9200?
Closed cheveyo20 closed 5 years ago
hjha287
commented
7 years ago By looking at the error it seems ES is not running on port 9300. Please check ES is running on what port and update the properties file accordingly. is it running on 9200?
cheveyo20
commented
7 years ago I deployed it like in the example on a different host than conductor:
version: "3.3"
services:
elasticsearch:
image: elasticsearch:2.4
ports:
- "9200:9200"
- "9300:9300"
networks:
- elk
kibana:
image: kibana:4.6
ports:
- "5601:5601"
networks:
- elk
networks:
elk:
driver: overlay
driver_opts:
encrypted: "true"REST-Port 9200 is answering me with: http://xx.xx.xx.xx:9200/_cluster/health?pretty
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}Kibana is also working and connecting. Telnet on port 9300 does work :)
Do i need something like an host suplier/tokensupplier for dynomite in this case? Or why am i getting UNKNOWN
cheveyo20
commented
7 years ago Ok you where right, i didnt know that elasticsearch does need the --net host option to be set. Now ES is working, but one error remains:
But in dynomite the keys were generated, so connection does work. Do i need something like an host suplier/tokensupplier for dynomite in this case?
6895 [pool-9-thread-1] ERROR com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - java.lang.RuntimeException: com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 1534224792
java.lang.RuntimeException: java.lang.RuntimeException: com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 1534224792
at com.netflix.dyno.queues.redis.RedisDynoQueue.pop(RedisDynoQueue.java:224)
at com.netflix.conductor.dao.dynomite.queue.DynoQueueDAO.pop(DynoQueueDAO.java:150)
at com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator.pollAndExecute(SystemTaskWorkerCoordinator.java:145)
at com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator.lambda$listen$1(SystemTaskWorkerCoordinator.java:125)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 1534224792
at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:603)
at com.netflix.dyno.queues.redis.RedisDynoQueue.execute(RedisDynoQueue.java:585)
at com.netflix.dyno.queues.redis.RedisDynoQueue.peekIds(RedisDynoQueue.java:512)
at com.netflix.dyno.queues.redis.RedisDynoQueue.prefetchIds(RedisDynoQueue.java:244)
at com.netflix.dyno.queues.redis.RedisDynoQueue.pop(RedisDynoQueue.java:216)
more
Caused by: com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 1534224792
at com.netflix.dyno.connectionpool.impl.hash.BinarySearchTokenMapper.getToken(BinarySearchTokenMapper.java:68)
at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getTokenForKey(TokenAwareSelection.java:110)
at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getPoolForOperation(TokenAwareSelection.java:73)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getFallbackHostPool(HostSelectionWithFallback.java:210)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:154)
at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnectionUsingRetryPolicy(HostSelectionWithFallback.java:121)
at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:292)
at com.netflix.dyno.jedis.DynoJedisClient.d_zrangeByScore(DynoJedisClient.java:2070)
at com.netflix.dyno.jedis.DynoJedisClient.zrangeByScore(DynoJedisClient.java:2065)
at com.netflix.dyno.queues.redis.RedisDynoQueue.lambda$peekIds$12(RedisDynoQueue.java:514)
at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:592)
more
7290 [qtp1682619279-32] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding system task DECISION
7291 [qtp1682619279-32] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding system task FORK
7291 [qtp1682619279-32] INFO com.netflix.conductor.core.execution.tasks.SystemTaskWorkerCoordinator - Adding system task JOIN
7385 [qtp1682619279-32] INFO com.netflix.dyno.queues.redis.RedisDynoQueue - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve task_1
7451 [qtp1682619279-32] INFO com.netflix.dyno.queues.redis.RedisDynoQueue - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve _deciderQueue
7458 [main] INFO com.netflix.conductor.server.ConductorServer - Kitchen sink workflows are created!I am not conductor SME's to comment on tokensupplier but I have used conductor with default tokensupplier implementation. I had encountered this error whenever the first dynomite node configured in the config file was down.
saidatta
commented
6 years ago I finally fixed it! I took the hint of changing it to network_mode: host. But, I had to add a condittion of where conductor-server dependencies should be successfully started before starting the condcutor-server
I did that using depends_on and condition: service_healthy.
kishorebanala
commented
5 years ago Closing this issue, as there is no activity for a while. Please feel free to open another issue if you still have questions.
avila3a
commented
5 years ago @saidatta
I finally fixed it! I took the hint of changing it to
network_mode: host. But, I had to add a condittion of where conductor-server dependencies should be successfully started before starting the condcutor-serverI did that using
depends_onandcondition: service_healthy.
Hiii I have this same problem but I don't understand how to solve it by adding depends_on and condition: service_healthy to which file (driver.properties). or where the properties are added
saidatta
commented
5 years ago @avila3a depends_on & condition is for the docker-compose file. point the pre-reqs services to conductor-server service.
cchristopherdxc
commented
4 years ago C:\DXC\Software\conductor\server>../gradlew server '..' is not recognized as an internal or external command, operable program or batch file.
i installed gradle and tried to execute the above command to startup conductor server. but it fails i tried run the server from root directory as below C:\DXC\Software\conductor>gradlew server
Configure project : Inferred project: conductor, version: 2.31.0-SNAPSHOT Publication nebula not found in project :.
Task :conductor-common:compileJava Note: C:\DXC\Software\conductor\common\src\main\java\com\netflix\conductor\common\utils\JsonMapperProvider.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: C:\DXC\Software\conductor\common\src\main\java\com\netflix\conductor\common\utils\ConstraintParamUtil.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details.
Task :conductor-common:jar FAILED
FAILURE: Build failed with an exception.
What went wrong: java.io.IOException: Unable to delete file: C:\DXC\Software\conductor\common\build\libs\conductor-common-2.31.0-SNAPSHOT.jar
Unable to delete file: C:\DXC\Software\conductor\common\build\libs\conductor-common-2.31.0-SNAPSHOT.jar
Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0. See https://docs.gradle.org/4.8.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 3s 3 actionable tasks: 3 executed
Hi, I have configured a Dynomite Cluster with 3 nodes, which seem to work (redis-cli to 8102 does work and keys are getting replicated), i use elasticsearch 2.4.6 with one node. The moment i start conductor, i get this:
So it does seem to be an issue with elasticsearch... but elasticsearch seems to work!!
The moment i run the kitchensink worker i get this on worker side:
I am not running on AWS, but i used naming convention..