Summary - Conductor Server

[kymr]

Link

• https://netflix.github.io/conductor/server/

[kymr]

Installing

Requirements

• Database: Dynomite
• Indexing Backend: Elasticsearch 2.x
• Servlet Container: Tomcat, Jetty, or similar running JDK 1.8 or higher

Build from Source

• To build from source, checkout the code from github and build server module using gradle build command.

• If you do not have gradle installed, you can run the command ./gradlew build from the project root.

• This produces conductor-server-all-VERSION.jar in the folder ./server/build/libs/

• The jar can be executed using:

  java -jar conductor-server-VERSION-all.jar

[kymr]

Configuration

• Conductor server uses a property file based configuration.

• The property file is passed to the Main class as a command line argument.

  java -jar conductor-server-all-VERSION.jar [PATH TO PROPERTY FILE] [log4j.properties file path]

• log4j.properties file path is optional and allows finer control over the logging (defaults to INFO level logging in the console).

Configuration Parameters

# Database persistence model.  Possible values are memory, redis, redis_cluster and dynomite.
# If omitted, the persistence used is memory
#
# memory : The data is stored in memory and lost when the server dies.  Useful for testing or demo
# redis : non-Dynomite based redis instance
# redis_cluster: AWS Elasticache Redis (cluster mode enabled).See [http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/Clusters.Create.CON.RedisCluster.html]
# dynomite : Dynomite cluster.  Use this for HA configuration.
db=dynomite

# Dynomite Cluster details.
# format is host:port:rack separated by semicolon
# for AWS Elasticache Redis (cluster mode enabled) the format is configuration_endpoint:port:us-east-1e. The region in this case does not matter
workflow.dynomite.cluster.hosts=host1:8102:us-east-1c;host2:8102:us-east-1d;host3:8102:us-east-1e

# Dynomite cluster name
workflow.dynomite.cluster.name=dyno_cluster_name

# Maximum connections to redis/dynomite
workflow.dynomite.connection.maxConnsPerHost=31

# Namespace for the keys stored in Dynomite/Redis
workflow.namespace.prefix=conductor

# Namespace prefix for the dyno queues
workflow.namespace.queue.prefix=conductor_queues

# No. of threads allocated to dyno-queues (optional)
queues.dynomite.threads=10

# Non-quorum port used to connect to local redis.  Used by dyno-queues.
# When using redis directly, set this to the same port as redis server
# For Dynomite, this is 22122 by default or the local redis-server port used by Dynomite.
queues.dynomite.nonQuorum.port=22122

# Transport address to elasticsearch
workflow.elasticsearch.url=localhost:9300

# Name of the elasticsearch cluster
workflow.elasticsearch.index.name=conductor

# Additional modules (optional)
conductor.additional.modules=class_extending_com.google.inject.AbstractModule

[kymr]

High Availability Configuration

• Conductor servers are stateless and can be deployed on multiple servers to handle scale and availability needs.
• The scalability of the server is achieved by scaling the Dynomite cluster along with dyno-queues which is used for queues.
• Clients connects to the server via HTTP load balancer or using Discovery (on NetflixOSS stack).

[kymr]

Check it

Dependencies

• How to replace database to mysql or others?
• Indexing backend ES2? or ES5? or others?
  • how to choose it?
  • how to install or configure it?
  • operation

Configuration

• redis cluster
  • nbase-arc?
  • or redis cluster
  • what is proper version?
  • only support AWS Elasticcache Redis, not common redis-cluster?
• max-connection
  • proper configuration
• Namespace
  • just config? or must set before config?
• dyno queues
  • how to replace it?
  • or is it just queue implementation on redis?
• nonQuorum port
  • I do not understand exactly what this value is for and how it works.
• es cluster address config
• Additional modules

High Availability Configuration

• redis-cluster settings
  • What is the right cluster size?
• ES
  • not effect on scalability?
  • indexing speed issues & capacity
• How to configure service discovery?