search

Netflix / conductor

Conductor is a microservices orchestration engine.
Apache License 2.0
12.83k stars 2.34k forks source link

Can you show me how to use Event Task and Event Handler? #125

Closed lydialmr120 closed 7 years ago

lydialmr120 commented 7 years ago

I can't trigger the event with my code.

Event Handler:

{
  "name": "evt_handler_test",
  "event": "conductor:startdynflow",
  "condition": "$.ref_flow !== null || $.ref_flow !== undefined || $.ref_flow !== ''",
  "actions": [
    {
      "action": "start_workflow",
      "start_workflow": {
        "name": "$.ref_flow",
        "version": 1,
        "correlationId": "string"
      },
      "expandInlineJSON": true
    }
  ],
  "active": true
}

Workflow with Event Task:

{
  "updateTime": 1489568993591,
  "name": "evt_test",
  "description": "A Simple workflow with 1 event task",
  "version": 1,
  "tasks": [
    {
      "name": "task_5",
      "taskReferenceName": "task_5",
      "inputParameters": {
        "sink": "conductor:startdynflow",
        "ref_flow": "${workflow.input.ref_flow}"
      },
      "type": "Event",
      "startDelay": 0
    }
  ],
  "schemaVersion": 2
}

Start workflow with

{"ref_flow":"sub_flow_1",
 "ref_task":"task_5"}

After this,the Event Task didn't create any event. The evt_test workflow is started as follow:

{
   "createTime": 1489569041964,
   "status": "RUNNING",
   "endTime": 0,
   "workflowId": "57602859-e963-4e27-942c-73c841c15c8f",
   "tasks": [
      {
         "taskType": "Event",
         "status": "SCHEDULED",
         "inputData": {
            "sink": "conductor:startdynflow",
            "ref_flow": "sub_flow_1"
         },
         "referenceTaskName": "task_5",
         "retryCount": 0,
         "seq": 1,
         "pollCount": 0,
         "taskDefName": "task_5",
         "scheduledTime": 1489569042064,
         "startTime": 0,
         "endTime": 0,
         "updateTime": 1489569042069,
         "startDelayInSeconds": 0,
         "retried": false,
         "callbackFromWorker": true,
         "responseTimeoutSeconds": 0,
         "workflowInstanceId": "57602859-e963-4e27-942c-73c841c15c8f",
         "taskId": "21a0ca60-719c-4af4-9b2a-e199e48d3081",
         "callbackAfterSeconds": 0,
         "queueWaitTime": 0,
         "taskStatus": "SCHEDULED"
      }
   ],
   "input": {
      "ref_flow": "sub_flow_1",
      "ref_task": "task_5"
   },
   "workflowType": "evt_test",
   "version": 1,
   "schemaVersion": 0,
   "startTime": 1489569041964
}
v1r3n commented 7 years ago

@lydialmr120 in the workflow definition, use the upper case for task type:

"type": "EVENT",
lydialmr120 commented 7 years ago

@v1r3n Thank you for your reply, I have modified the type to upper case, and the event task is executed, but the event handler didn't execute the action, I am not sure whether it receive the event. Since the event task's output is conductor:evt_test:task_1, I have modified the event handler definition:

{
  "name": "evt_handler_test",
  "event": "conductor:evt_test:task_1",
  "condition": "1>0",
  "actions": [
    {
      "action": "start_workflow",
      "start_workflow": {
        "name": "sub_flow_1",
        "version": 1
      },
      "expandInlineJSON": true
    }
  ],
 "active":true
}
v1r3n commented 7 years ago

@lydialmr120 the event is listening on the "task_1", can you update the event handler as below:

"event": "conductor:evt_test:task_5"

The format is: conductor:<workflowname>:<task_refname>

Also, try by removing condition to make sure it is wired up correctly and then add the condition?

lydialmr120 commented 7 years ago

The work task_refname is just task_1. After modifing the event type,the new workflow JSON is as follow:

{
   "createTime": 1489737457583,
   "updateTime": 1489737457847,
   "status": "COMPLETED",
   "endTime": 1489737457847,
   "workflowId": "39dc793b-521b-4817-a01a-f426d3021a64",
   "tasks": [
      {
         "taskType": "EVENT",
         "status": "COMPLETED",
         "inputData": {
            "ref_flow": "sub_flow_1",
            "sink": "conductor"
         },
         "referenceTaskName": "task_1",
         "retryCount": 0,
         "seq": 1,
         "pollCount": 0,
         "taskDefName": "EVENT",
         "scheduledTime": 1489737457660,
         "startTime": 1489737457725,
         "endTime": 1489737457729,
         "updateTime": 1489737457729,
         "startDelayInSeconds": 0,
         "retried": false,
         "callbackFromWorker": true,
         "responseTimeoutSeconds": 0,
         "workflowInstanceId": "39dc793b-521b-4817-a01a-f426d3021a64",
         "taskId": "b7819965-79bd-44ee-b587-720c237cb59b",
         "callbackAfterSeconds": 0,
         "outputData": {
            "event_produced": "conductor:evt_test:task_1"
         },
         "queueWaitTime": 65,
         "taskStatus": "COMPLETED"
      }
   ],
   "input": {
      "ref_flow": "sub_flow_1",
      "ref_task": "task_5"
   },
   "output": {
      "event_produced": "conductor:evt_test:task_1"
   },
   "workflowType": "evt_test",
   "version": 1,
   "schemaVersion": 2,
   "startTime": 1489737457583
}
lydialmr120 commented 7 years ago

I also remove the condition, but the event handler still can't execute the action.

{
  "name": "evt_handler_test",
  "event": "conductor:evt_test:task_1",
  "actions": [
    {
      "action": "start_workflow",
      "start_workflow": {
        "name": "sub_flow_1",
        "version": 1
      },
      "expandInlineJSON": true
    }
  ],
  "active": true
}
v1r3n commented 7 years ago

@lydialmr120 I was able to trigger the sub-workflow correctly with the following workflow and event handler definition:

Workflow Definition

{
    "name": "evt_test",
    "description": "A Simple workflow with 1 event task",
    "version": 1,
    "tasks": [
        {
            "name": "task_1",
            "taskReferenceName": "task_1",
            "inputParameters": {
                "sink": "conductor",
                "ref_flow": "${workflow.input.ref_flow}"
            },
            "sink": "conductor",
            "type": "EVENT",
            "startDelay": 0
        }
    ],
    "schemaVersion": 2
}

Event Handler

{
    "name": "evt_handler_test",
    "event": "conductor:evt_test:task_1",
    "actions": [
        {
            "action": "start_workflow",
            "start_workflow": {
                "name": "sub_flow_1",
                "version": 1
            },
            "expandInlineJSON": true
        }
    ],
    "active": true
}

once the event handler is created, it can take few seconds up to a minute for the background thread to start polling for it. Do you see any errors in the log?

lydialmr120 commented 7 years ago

No errors

Using conductor-conf.properties
0    [main] INFO  com.netflix.dyno.jedis.DynoJedisClient  - Starting connection pool for app conductor
19   [pool-3-thread-1] INFO  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - Adding host connection pool for host: Host [hostname=192.168.35.213, ipAddress=null, port=8102, rack: us-east-1b, datacenter: us-east-1, status: Up]
19   [pool-3-thread-1] INFO  com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl  - Priming connection pool for host:Host [hostname=192.168.35.213, ipAddress=null, port=8102, rack: us-east-1b, datacenter: us-east-1, status: Up], with conns:3
125  [pool-3-thread-1] INFO  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - Successfully primed 3 of 3 to Host [hostname=192.168.35.213, ipAddress=null, port=8102, rack: us-east-1b, datacenter: us-east-1, status: Up]
168  [main] INFO  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - registered mbean com.netflix.dyno.connectionpool.impl:type=MonitorConsole
174  [main] INFO  com.netflix.conductor.server.ConductorServer  - Starting conductor server using dynomite cluster dyn_o_mite

                     _            _             
  ___ ___  _ __   __| |_   _  ___| |_ ___  _ __ 
 / __/ _ \| '_ \ / _` | | | |/ __| __/ _ \| '__|
| (_| (_) | | | | (_| | |_| | (__| || (_) | |   
 \___\___/|_| |_|\__,_|\__,_|\___|\__\___/|_|   

647  [main] INFO  com.netflix.conductor.dao.dynomite.queue.DynoQueueDAO  - DynoQueueDAO initialized with prefix conductor_queues.test!
2175 [main] INFO  com.netflix.conductor.contribs.http.HttpTask  - HttpTask initialized...
3086 [main] INFO  org.elasticsearch.plugins  - [Supercharger] modules [], plugins [], sites []
4612 [main] INFO  org.eclipse.jetty.util.log  - Logging initialized @8024ms
4743 [main] INFO  org.eclipse.jetty.server.Server  - jetty-9.3.9.v20160517
三月 21, 2017 10:55:34 上午 com.sun.jersey.api.core.PackagesResourceConfig init
信息: Scanning for root resource and provider classes in the packages:
  com.netflix.conductor.server.resources
  io.swagger.jaxrs.json
  io.swagger.jaxrs.listing
三月 21, 2017 10:55:34 上午 com.sun.jersey.api.core.ScanningResourceConfig logClasses
信息: Root resource classes found:
  class io.swagger.jaxrs.listing.ApiListingResource
  class com.netflix.conductor.server.resources.TaskResource
  class com.netflix.conductor.server.resources.WorkflowResource
  class com.netflix.conductor.server.resources.EventResource
  class io.swagger.jaxrs.listing.AcceptHeaderApiListingResource
  class com.netflix.conductor.server.resources.MetadataResource
  class com.netflix.conductor.server.resources.AdminResource
三月 21, 2017 10:55:34 上午 com.sun.jersey.api.core.ScanningResourceConfig logClasses
信息: Provider classes found:
  class io.swagger.jaxrs.listing.SwaggerSerializers
  class com.netflix.conductor.server.resources.GenericExceptionMapper
  class com.netflix.conductor.server.resources.WebAppExceptionMapper
  class com.netflix.conductor.server.resources.ApplicationExceptionMapper
三月 21, 2017 10:55:34 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
信息: Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
三月 21, 2017 10:55:34 上午 com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
信息: Initiating Jersey application, version 'Jersey: 1.19.1 03/11/2016 02:42 PM'
三月 21, 2017 10:55:34 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.ApplicationExceptionMapper to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:34 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.GenericExceptionMapper to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:34 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
5149 [RxComputationScheduler-1] INFO  com.netflix.dyno.queues.redis.RedisDynoQueue  - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve evt_test:task_1
三月 21, 2017 10:55:34 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.AdminResource to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:35 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.EventResource to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:35 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.MetadataResource to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:35 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.TaskResource to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:35 上午 com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
信息: Binding com.netflix.conductor.server.resources.WorkflowResource to GuiceInstantiatedComponentProvider
三月 21, 2017 10:55:35 上午 com.sun.jersey.spi.inject.Errors processErrorMessages
警告: The following warnings have been detected with resource and/or provider classes:
  WARNING: Return type java.util.Map<java.lang.String, ?> of method public java.util.Map<java.lang.String, ?> com.netflix.conductor.server.resources.EventResource.getEventQueues(boolean) is not resolvable to a concrete type
5626 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler  - Started o.e.j.s.ServletContextHandler@1e86a5a7{/,file:///E:/SpringCloud/Project/conductor/server/build/resources/main/swagger-ui/,AVAILABLE}
5662 [main] INFO  org.eclipse.jetty.server.AbstractConnector  - Started ServerConnector@2a2ef072{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
5662 [main] INFO  org.eclipse.jetty.server.Server  - Started @9078ms
Started server on http://localhost:8080/
283883 [qtp582702662-37] INFO  com.netflix.dyno.queues.redis.RedisDynoQueue  - com.netflix.dyno.queues.redis.RedisDynoQueue is ready to serve _deciderQueue
lydialmr120 commented 7 years ago

I added some print in the code, find it can't get any message from the dynoqueue, but there is data in redis. I added print as follow:

@Override
public List<com.netflix.conductor.core.events.queue.Message> pollMessages(String queueName, int count, int timeout) {
        List<Message> msgs = queues.get(queueName).pop(count, timeout, TimeUnit.MILLISECONDS);
        System.out.println(queueName+" pollMsg size:"+msgs.size());
        return msgs.stream().map(msg -> new com.netflix.conductor.core.events.queue.Message(msg.getId(), msg.getPayload(), null)).collect(Collectors.toList());
    }

And the output is 0.

evt_test:task_1 pollMsg size:0

View the data in redis through cmd.

192.168.35.213:8102> zrange conductor_queues.test.QUEUE.evt_test:task_1.b 0 -1
1) "c876d75f-60a0-4f9b-a3ff-e40d2761ec61"
2) "3885facc-c019-454b-976c-acbffb0f7c0e"
3) "05721f42-d857-48a5-ab97-5512e02dfc2d"
v1r3n commented 7 years ago

Can you share your conductor-conf.properties file? what is the value of EC2_AVAILABILTY_ZONE property in your config file?

lydialmr120 commented 7 years ago 
# Database persistence model.  Possible values are memory, redis, and dynomite.
# If ommitted, the persistence used is memory
#
# memory : The data is stored in memory and lost when the server dies.  Useful for testing or demo
# redis : non-Dynomite based redis instance
# dynomite : Dynomite cluster.  Use this for HA configuration.
db=dynomite

# Dynomite Cluster details.
# format is host:port:rack separated by semicolon
workflow.dynomite.cluster.hosts=192.168.35.213:8102:us-east-1b

# Dynomite cluster name
workflow.dynomite.cluster.name=dyn_o_mite

# Namespace for the keys stored in Dynomite/Redis
workflow.namespace.prefix=conductor

# Namespace prefix for the dyno queues
workflow.namespace.queue.prefix=conductor_queues

# No. of threads allocated to dyno-queues (optional)
queues.dynomite.threads=10

# Non-quorum port used to connect to local redis.  Used by dyno-queues.
# When using redis directly, set this to the same port as redis server
# For Dynomite, this is 22122 by default or the local redis-server port used by Dynomite.
queues.dynomite.nonQuorum.port=22122

# Transport address to elasticsearch
workflow.elasticsearch.url=192.168.35.214:9300

# Name of the elasticsearch cluster
workflow.elasticsearch.index.name=conductor

# For a single node dynomite or redis server, make sure the value below is set to same as rack specified in the "workflow.dynomite.cluster.hosts" property. 
EC2_AVAILABILTY_ZONE=us-east-1b
v1r3n commented 7 years ago

@lydialmr120 can you confirm the workflow and event definitions you have is as follows?

Workflow Definition

{
    "name": "evt_test",
    "description": "A Simple workflow with 1 event task",
    "version": 1,
    "tasks": [
        {
            "name": "task_1",
            "taskReferenceName": "task_1",
            "inputParameters": {
                "sink": "conductor",
                "ref_flow": "${workflow.input.ref_flow}"
            },
            "sink": "conductor",
            "type": "EVENT",
            "startDelay": 0
        }
    ],
    "schemaVersion": 2
}

Event Handler

{
    "name": "evt_handler_test",
    "event": "conductor:evt_test:task_1",
    "actions": [
        {
            "action": "start_workflow",
            "start_workflow": {
                "name": "sub_flow_1",
                "version": 1
            },
            "expandInlineJSON": true
        }
    ],
    "active": true
}
lydialmr120 commented 7 years ago

Yes, I have confirmed it.

lydialmr120 commented 7 years ago

Today I modify the dynomite config, and rerun this demo,I get some error message.

281290 [pool-4-thread-5] WARN  com.netflix.dyno.jedis.JedisConnectionFactory  - Caught RuntimeException: ERR syntax error
281292 [pool-4-thread-5] WARN  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - Received FatalConnectionException; closing connection {port=9102, host=192.168.35.211} to host Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up]
281294 [pool-4-thread-5] ERROR com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl  - Failed to close connection for host: Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up] Unexpected end of stream.
281304 [pool-4-thread-6] WARN  com.netflix.dyno.jedis.JedisConnectionFactory  - Caught RuntimeException: ERR syntax error
281304 [pool-4-thread-6] WARN  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - Received FatalConnectionException; closing connection {port=9102, host=192.168.35.211} to host Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up]
281306 [pool-4-thread-6] ERROR com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl  - Failed to close connection for host: Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up] Unexpected end of stream.
281314 [pool-4-thread-9] WARN  com.netflix.dyno.jedis.JedisConnectionFactory  - Caught RuntimeException: ERR syntax error
281314 [pool-4-thread-9] WARN  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - Received FatalConnectionException; closing connection {port=9102, host=192.168.35.211} to host Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up]
281315 [pool-4-thread-9] ERROR com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl  - Failed to close connection for host: Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up] Unexpected end of stream.
Exception in thread "RxComputationScheduler-1" java.lang.IllegalStateException: Exception thrown on Scheduler.Worker thread. Add `onError` handling.
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:57)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: rx.exceptions.OnErrorNotImplementedException: com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at rx.internal.util.InternalObservableUtils$ErrorNotImplementedAction.call(InternalObservableUtils.java:386)
    at rx.internal.util.InternalObservableUtils$ErrorNotImplementedAction.call(InternalObservableUtils.java:383)
    at rx.internal.util.ActionSubscriber.onError(ActionSubscriber.java:44)
    at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:153)
    at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:115)
    at com.netflix.conductor.core.events.queue.dyno.DynoObservableQueue$1.lambda$call$2(DynoObservableQueue.java:120)
    at rx.internal.util.ActionSubscriber.onError(ActionSubscriber.java:44)
    at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:153)
    at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:115)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:818)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:579)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onError(OperatorMerge.java:276)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
    at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52)
    at rx.Scheduler$Worker$1.call(Scheduler.java:137)
    at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    ... 7 more
Caused by: java.lang.RuntimeException: com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:611)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:608)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:608)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.execute(RedisDynoQueue.java:595)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.pop(RedisDynoQueue.java:216)
    at com.netflix.conductor.dao.dynomite.queue.DynoQueueDAO.pollMessages(DynoQueueDAO.java:160)
    at com.netflix.conductor.core.events.queue.dyno.DynoObservableQueue.receiveMessages(DynoObservableQueue.java:103)
    at com.netflix.conductor.core.events.queue.dyno.DynoObservableQueue$1.lambda$call$0(DynoObservableQueue.java:114)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
    ... 11 more
Caused by: com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at com.netflix.dyno.jedis.JedisConnectionFactory$JedisConnection.execute(JedisConnectionFactory.java:105)
    at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:298)
    at com.netflix.dyno.jedis.DynoJedisClient.d_zadd(DynoJedisClient.java:1653)
    at com.netflix.dyno.jedis.DynoJedisClient.zadd(DynoJedisClient.java:1648)
    at com.netflix.dyno.queues.redis.RedisDynoQueue._pop(RedisDynoQueue.java:278)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.lambda$pop$7(RedisDynoQueue.java:232)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more
Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at redis.clients.jedis.Protocol.processError(Protocol.java:117)
    at redis.clients.jedis.Protocol.process(Protocol.java:151)
    at redis.clients.jedis.Protocol.read(Protocol.java:205)
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
    at redis.clients.jedis.Connection.getIntegerReply(Connection.java:222)
    at redis.clients.jedis.Jedis.zadd(Jedis.java:1341)
    at com.netflix.dyno.jedis.DynoJedisClient$91.execute(DynoJedisClient.java:1657)
    at com.netflix.dyno.jedis.DynoJedisClient$91.execute(DynoJedisClient.java:1653)
    at com.netflix.dyno.jedis.JedisConnectionFactory$JedisConnection.execute(JedisConnectionFactory.java:86)
    ... 9 more
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: 275
    at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:109)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
    at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52)
    at rx.Scheduler$Worker$1.call(Scheduler.java:137)
    at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    ... 3 more

My server config

# Database persistence model.  Possible values are memory, redis, and dynomite.
# If ommitted, the persistence used is memory
#
# memory : The data is stored in memory and lost when the server dies.  Useful for testing or demo
# redis : non-Dynomite based redis instance
# dynomite : Dynomite cluster.  Use this for HA configuration.
db=dynomite

# Dynomite Cluster details.
# format is host:port:rack separated by semicolon
workflow.dynomite.cluster.hosts=192.168.35.211:9102:us-east-1a;192.168.35.212:9102:us-east-1c

# Dynomite cluster name
workflow.dynomite.cluster.name=dyn_o_mite

# Namespace for the keys stored in Dynomite/Redis
workflow.namespace.prefix=conductor

# Namespace prefix for the dyno queues
workflow.namespace.queue.prefix=conductor_queues

# No. of threads allocated to dyno-queues (optional)
queues.dynomite.threads=10

# Non-quorum port used to connect to local redis.  Used by dyno-queues.
# When using redis directly, set this to the same port as redis server
# For Dynomite, this is 22122 by default or the local redis-server port used by Dynomite.
queues.dynomite.nonQuorum.port=22122

# Transport address to elasticsearch
workflow.elasticsearch.url=192.168.35.212:9301
#workflow.elasticsearch.url=192.168.35.214:9300

# Name of the elasticsearch cluster
workflow.elasticsearch.index.name=conductor

# For a single node dynomite or redis server, make sure the value below is set to same as rack specified in the "workflow.dynomite.cluster.hosts" property. 
EC2_AVAILABILTY_ZONE=us-east-1c

How to config EC2_AVAILABILTY_ZONE when I uses two node as redis server?

lydialmr120 commented 7 years ago

I have resolved ERR syntax error, I think the server.properties is wrong with EC2_AVAILABILTY_ZONE, the Correct spelling is EC2_AVAILABILITY_ZONE. But I get another error, how to solve it?

66255 [qtp2090894907-39] ERROR com.netflix.conductor.service.ExecutionService  - NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 3957939958
com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 3957939958
    at com.netflix.dyno.connectionpool.impl.hash.BinarySearchTokenMapper.getToken(BinarySearchTokenMapper.java:68)
    at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getTokenForKey(TokenAwareSelection.java:110)
    at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getPoolForOperation(TokenAwareSelection.java:73)
    at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getFallbackHostPool(HostSelectionWithFallback.java:209)
    at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:153)
    at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnectionUsingRetryPolicy(HostSelectionWithFallback.java:120)
    at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:292)
    at com.netflix.dyno.jedis.DynoJedisClient.d_get(DynoJedisClient.java:340)
    at com.netflix.dyno.jedis.DynoJedisClient.get(DynoJedisClient.java:334)
    at com.netflix.conductor.dao.dynomite.DynoProxy.get(DynoProxy.java:89)
    at com.netflix.conductor.dao.dynomite.RedisExecutionDAO.getWorkflow(RedisExecutionDAO.java:277)
    at com.netflix.conductor.service.ExecutionService.lambda$search$1(ExecutionService.java:288)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
    at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
    at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool.helpComplete(ForkJoinPool.java:1870)
    at java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:2467)
    at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:324)
    at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
    at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
    at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.netflix.conductor.service.ExecutionService.search(ExecutionService.java:295)
    at com.netflix.conductor.server.resources.WorkflowResource.search(WorkflowResource.java:223)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
    at com.netflix.conductor.server.JerseyModule$1.doFilter(JerseyModule.java:99)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:524)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:745)
66324 [ForkJoinPool.commonPool-worker-2] ERROR com.netflix.conductor.service.ExecutionService  - NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 3957939958
com.netflix.dyno.connectionpool.exception.NoAvailableHostsException: NoAvailableHostsException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=0]Token not found for key hash: 3957939958
    at com.netflix.dyno.connectionpool.impl.hash.BinarySearchTokenMapper.getToken(BinarySearchTokenMapper.java:68)
    at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getTokenForKey(TokenAwareSelection.java:110)
    at com.netflix.dyno.connectionpool.impl.lb.TokenAwareSelection.getPoolForOperation(TokenAwareSelection.java:73)
    at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getFallbackHostPool(HostSelectionWithFallback.java:209)
    at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:153)
    at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnectionUsingRetryPolicy(HostSelectionWithFallback.java:120)
    at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:292)
    at com.netflix.dyno.jedis.DynoJedisClient.d_get(DynoJedisClient.java:340)
    at com.netflix.dyno.jedis.DynoJedisClient.get(DynoJedisClient.java:334)
    at com.netflix.conductor.dao.dynomite.DynoProxy.get(DynoProxy.java:89)
    at com.netflix.conductor.dao.dynomite.RedisExecutionDAO.getWorkflow(RedisExecutionDAO.java:277)
    at com.netflix.conductor.service.ExecutionService.lambda$search$1(ExecutionService.java:288)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
    at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
    at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
v1r3n commented 7 years ago

Adding the Dynomite experts, to comment on this. @ipapapa @shailesh33 do you now what will cause this error?

Token not found for key hash: 3957939958
ipapapa commented 7 years ago

The problem is related to the Dyno not seeing the correct topology.

If the EC2_AVAILABILITY_ZONE environment property has not been set then the topology that Dyno sees may not be correct. We need to add that hook for OSS users that are not using AWS or do not have the variable set. More specifically when trying to getLocalZone, it will return null.

We had the issue with another OSS user who used:

 Host(dynoNode.getHostName(),dynoNode.getIp(),dynoNode.getPort(),dynoNode.getDc(),dynoNode.getDc(),Host.Status.Up);
lydialmr120 commented 7 years ago

I start the server with

java -DEC2_AVAILABILITY_ZONE=us-east-1c -jar build\libs\conductor-server-1.7.0-SNAPSHOT-all.jar conductor-conf.properties log4j.properties

Still get error when starting, even so , the Conductor started sucessful, but event handler can't execute action yet.

4320 [pool-4-thread-6] WARN  com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl  - Received FatalConnectionException; closing connection {port=9102, host=192.168.35.211} to host Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up]
4321 [pool-4-thread-6] ERROR com.netflix.dyno.connectionpool.impl.HostConnectionPoolImpl  - Failed to close connection for host: Host [hostname=192.168.35.211, ipAddress=null, port=9102, rack: us-east-1a, datacenter: us-east-1, status: Up] Unexpected end of stream.
Exception in thread "RxComputationScheduler-1" java.lang.IllegalStateException: Exception thrown on Scheduler.Worker thread. Add `onError` handling.
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:57)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: rx.exceptions.OnErrorNotImplementedException: com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at rx.internal.util.InternalObservableUtils$ErrorNotImplementedAction.call(InternalObservableUtils.java:386)
    at rx.internal.util.InternalObservableUtils$ErrorNotImplementedAction.call(InternalObservableUtils.java:383)
    at rx.internal.util.ActionSubscriber.onError(ActionSubscriber.java:44)
    at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:153)
    at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:115)
    at com.netflix.conductor.core.events.queue.dyno.DynoObservableQueue$1.lambda$call$2(DynoObservableQueue.java:120)
    at rx.internal.util.ActionSubscriber.onError(ActionSubscriber.java:44)
    at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:153)
    at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:115)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:818)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:579)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onError(OperatorMerge.java:276)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
    at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52)
    at rx.Scheduler$Worker$1.call(Scheduler.java:137)
    at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    ... 7 more
Caused by: java.lang.RuntimeException: com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:611)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:608)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.executeWithRetry(RedisDynoQueue.java:608)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.execute(RedisDynoQueue.java:595)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.pop(RedisDynoQueue.java:216)
    at com.netflix.conductor.dao.dynomite.queue.DynoQueueDAO.pollMessages(DynoQueueDAO.java:160)
    at com.netflix.conductor.core.events.queue.dyno.DynoObservableQueue.receiveMessages(DynoObservableQueue.java:103)
    at com.netflix.conductor.core.events.queue.dyno.DynoObservableQueue$1.lambda$call$0(DynoObservableQueue.java:114)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
    ... 11 more
Caused by: com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at com.netflix.dyno.jedis.JedisConnectionFactory$JedisConnection.execute(JedisConnectionFactory.java:105)
    at com.netflix.dyno.connectionpool.impl.ConnectionPoolImpl.executeWithFailover(ConnectionPoolImpl.java:298)
    at com.netflix.dyno.jedis.DynoJedisClient.d_zadd(DynoJedisClient.java:1653)
    at com.netflix.dyno.jedis.DynoJedisClient.zadd(DynoJedisClient.java:1648)
    at com.netflix.dyno.queues.redis.RedisDynoQueue._pop(RedisDynoQueue.java:278)
    at com.netflix.dyno.queues.redis.RedisDynoQueue.lambda$pop$7(RedisDynoQueue.java:232)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more
Caused by: redis.clients.jedis.exceptions.JedisDataException: ERR syntax error
    at redis.clients.jedis.Protocol.processError(Protocol.java:117)
    at redis.clients.jedis.Protocol.process(Protocol.java:151)
    at redis.clients.jedis.Protocol.read(Protocol.java:205)
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
    at redis.clients.jedis.Connection.getIntegerReply(Connection.java:222)
    at redis.clients.jedis.Jedis.zadd(Jedis.java:1341)
    at com.netflix.dyno.jedis.DynoJedisClient$91.execute(DynoJedisClient.java:1657)
    at com.netflix.dyno.jedis.DynoJedisClient$91.execute(DynoJedisClient.java:1653)
    at com.netflix.dyno.jedis.JedisConnectionFactory$JedisConnection.execute(JedisConnectionFactory.java:86)
    ... 9 more
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: 0
    at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:109)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
    at rx.internal.operators.OnSubscribeTimerPeriodically$1.call(OnSubscribeTimerPeriodically.java:52)
    at rx.Scheduler$Worker$1.call(Scheduler.java:137)
    at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    ... 3 more
v1r3n commented 7 years ago

What version of redis server are you using with dynomite? I see some errors above that seems to indicate it is using redis version below 3.0.7. Can you try and update the redis server to 3.0.7+ if not and try?

lydialmr120 commented 7 years ago

The redis version is 2.8.19, and I can't update it now.Is there any other solution?

v1r3n commented 7 years ago

@lydialmr120 we rely on the commands available in 3.0.7+ versions of Redis server. Using an older version of server will not work.

ipapapa commented 7 years ago

@v1r3n The Jedis client version in Dyno is 2.8.1, hence the corresponding supported commands. A Redis server ver 2.8.19 would work fine with the APIs. Similarly, according to the Jedis 2.8 Readme Redis 2.8.5 and above is supported.

Redis 3+ includes some clustering improvements, which are important for the warm up (in case a Dynomite node fails and needs to get data from a peer node).

Though, I see the weird error ERR syntax error which comes from Redis and makes me think that there is a misalignment in the versions as you suggest. @lydialmr120 can you create a simple Java app with JedisClient and see if it works with your Redis version?

lydialmr120 commented 7 years ago

@ipapapa I have tried it a simple JedisClient can work with my redis 2.8.19. @v1r3n I have changed redis version to 3.2.8, and start the Conductor server with -DEC2_AVAILABILITY_ZONE even I have specified it in properties, then all seem to be fine， the Event Handler executed the action too. Thank you for your help.@ipapapa @v1r3n