An exception occurred: Error polling system task in queue:[START_WORKFLOW|HTTP|JOIN...]

[zengqinglei]

Describe the bug When I start the conductor through docker-compose -f docker-compose.yaml -f docker-compose-dynomite.yaml up, frequent errors occur: Error polling system task in queue:[START_WORKFLOW|HTTP|JOIN...]

Details Conductor version:3.13.5 Persistence implementation: Dynomite Queue implementation: Dynomite Lock: NO

To Reproduce Steps to reproduce the behavior:

1. After pulling git clone {url} to clone the code, use the following command to build the conductor image

   cd docker/server
   docker build -f Dockerfile -t conductor-server:v3.13.5 ../../
   cd docker/ui
   docker build -f Dockerfile -t conductor-ui:v3.13.5 ../../

2. Define docker-compose.yaml

   
   version: '2.3'

services: conductor-server: environment:

• TZ=Asia/Shanghai
• CONFIG_PROP=config-local.properties image: conductor:server-v3.13.5 container_name: conductor-server networks:
• internal ports:
• 8080:8080 volumes:
• ./server/config/config-local.properties:/app/config/config-local.properties
• ./server/config/log4j2.xml:/app/config/log4j2.xml
• ./server/logs:/app/logs healthcheck: test: ["CMD", "curl","-I" ,"-XGET", "http://localhost:8080/health"] interval: 60s timeout: 30s retries: 12 links:
• elasticsearch:es

• redis:rs depends_on: elasticsearch: condition: service_healthy redis: condition: service_healthy logging: driver: "json-file" options: max-size: "1024m" max-file: "3"

  conductor-ui: environment:

• TZ=Asia/Shanghai
• WF_SERVER=http://conductor-server:8080 image: conductor-ui:v3.13.5 container_name: conductor-ui networks:
• internal ports:
• 5000:5000 links:

• conductor-server stdin_open: true

  redis: image: redis:6.2.3-alpine environment:

• TZ=Asia/Shanghai privileged: true networks:
• internal ports:

• 6379:6379 healthcheck: test: [ "CMD", "redis-cli","ping" ]

  elasticsearch: image: elasticsearch:6.8.15 container_name: elasticsearch environment:

• TZ=Asia/Shanghai
• "ES_JAVA_OPTS=-Xms512m -Xmx1024m"
• transport.host=0.0.0.0
• discovery.type=single-node
• xpack.security.enabled=false volumes:
• esdata-conductor:/usr/share/elasticsearch/data networks:
• internal ports:
• 9200:9200
• 9300:9300 healthcheck: test: wget http://localhost:9200/ -O /dev/null interval: 5s timeout: 5s retries: 12 logging: driver: "json-file" options: max-size: "1024m" max-file: "3"

volumes: esdata-conductor: driver: local

networks: internal:

4. Define docker-compose-dynomite.yaml
``` yaml
version: '2.3'

services:
  conductor-server:
    environment:
      - TZ=Asia/Shanghai
      - CONFIG_PROP=config.properties
    volumes:
      - ./server/config/config.properties:/app/config/config.properties
      - ./server/config/log4j2.xml:/app/config/log4j2.xml
      - ./server/logs:/app/logs
    links:
      - dynomite:dyno1
    depends_on:
      dynomite:
        condition: service_healthy

  dynomite:
    image: v1r3n/dynomite
    networks:
      - internal
    ports:
      - 8102:8102
    healthcheck:
      test: timeout 5 bash -c 'cat < /dev/null > /dev/tcp/localhost/8102'
      interval: 5s
      timeout: 5s
      retries: 12
    logging:
      driver: "json-file"
      options:
        max-size: "1024m"
        max-file: "3"

networks:
  internal:

5. Define config.properties

   
   # Servers.
   conductor.grpc-server.enabled=false

Database persistence type.

conductor.db.type=dynomite

Dynomite Cluster details.

format is host:port:rack separated by semicolon

conductor.redis.hosts=dyno1:8102:us-east-1c

Dynomite cluster name

conductor.redis.clusterName=dyno1

Namespace for the keys stored in Dynomite/Redis

conductor.redis.workflowNamespacePrefix=conductor

Namespace prefix for the dyno queues

conductor.redis.queueNamespacePrefix=conductor_queues

No. of threads allocated to dyno-queues (optional)

queues.dynomite.threads=10

By default with dynomite, we want the repairservice enabled

conductor.app.workflowRepairServiceEnabled=true

Non-quorum port used to connect to local redis. Used by dyno-queues.

When using redis directly, set this to the same port as redis server

For Dynomite, this is 22122 by default or the local redis-server port used by Dynomite.

conductor.redis.queuesNonQuorumPort=22122

Elastic search instance indexing is enabled.

conductor.indexing.enabled=true

Transport address to elasticsearch

conductor.elasticsearch.url=http://es:9200

Name of the elasticsearch cluster

conductor.elasticsearch.indexName=conductor

conductor.event-queues.amqp.queueType=classic

conductor.event-queues.amqp.sequentialMsgProcessing=true

Additional modules for metrics collection exposed via logger (optional)

conductor.metrics-logger.enabled=true

conductor.metrics-logger.reportPeriodSeconds=15

Additional modules for metrics collection exposed to Prometheus (optional)

conductor.metrics-prometheus.enabled=true

management.endpoints.web.exposure.include=prometheus

To enable Workflow/Task Summary Input/Output JSON Serialization, use the following:

conductor.app.summary-input-output-json-serialization.enabled=true

Load sample kitchen sink workflow

loadSample=false

conductor.elasticsearch.clusterHealthColor=yellow

logging.config=/app/config/log4j2.xml

6. Define log4j2.xml
``` xml
<Configuration status="WARN">
    <Appenders>
        <Console name="CONSOLE">
            <PatternLayout pattern="%d{ISO8601} %highlight{%-5level }[%style{%t}{bright,blue}] %style{%C{1.}}{bright,yellow}: %msg%n%throwable"/>
        </Console>
    </Appenders>

    <Loggers>
        <Root level="INFO">
            <AppenderRef ref="CONSOLE" />
        </Root>
    </Loggers>
</Configuration>

7. Start up：docker-compose -f docker-compose.yaml -f docker-compose-dynomite.yaml up
8. See error

But through the UI, I can still add workflows, but not run workflows

When I start conductor via docker-compose up , everything works fine.

Looking forward to your reply as soon as possible, thank you!

[BradenEads]

It seems like there might be an issue with the Dynomite configuration in your setup. Here are a few suggestions to troubleshoot and potentially fix the problem:

Check Dynomite logs: Inspect the logs of the Dynomite container to see if there are any errors or issues with the configuration. You can view the logs using the following command:

docker logs

It seems like there might be an issue with the Dynomite configuration in your setup. Here are a few suggestions to troubleshoot and potentially fix the problem:

Check Dynomite logs: Inspect the logs of the Dynomite container to see if there are any errors or issues with the configuration. You can view the logs using the following command: bash

docker logs <dynomite_container_id>

Replace with the actual container ID of your Dynomite instance.

Verify Dynomite configuration: Make sure that the Dynomite configuration is correct and all required properties are set. You can refer to the official Dynomite documentation for more information on how to configure Dynomite.

Test Dynomite connectivity: Check if Conductor can connect to the Dynomite instance by running a simple command from within the Conductor container:

docker exec -it <conductor_container_id> redis-cli -h dyno1 -p 8102 ping

Replace with the actual container ID of your Conductor instance. This command should return PONG if the connection is successful.

Test Dynomite cluster: Make sure that the Dynomite cluster is properly set up and running. If you are using a single-node setup for testing purposes, ensure that the dynomite service in the docker-compose-dynomite.yaml file has the correct configuration.

Inspect Conductor logs: Review the logs of the Conductor container for any error messages or issues related to Dynomite connectivity or configuration. You can view the logs using the following command:

docker logs <conductor_container_id>

Replace with the actual container ID of your Conductor instance.

By following these steps, you should be able to identify the root cause of the issue and resolve it. If the issue persists, please provide more information about your setup, any error messages, and logs to help diagnose the problem.

[zengqinglei]

@BradenEads I passed docker logs -f dynomite, and stopped the conductor-server, the log content of dynomite is as follows:

I did not find the error from the log, my startup container docker-compose.yaml is as follows, can you help analyze the following:

version: '2.3'

services:
  dynomite:
    environment:
      - TZ=Asia/Shanghai
    container_name: dynomite
    image: v1r3n/dynomite
    networks:
      - internal
    ports:
      - 8102:8102
    healthcheck:
      test: timeout 5 bash -c 'cat < /dev/null > /dev/tcp/localhost/8102'
      interval: 5s
      timeout: 5s
      retries: 12
    logging:
      driver: "json-file"
      options:
        max-size: "1024m"
        max-file: "3"

[lijia-rengage]

I deployed dynomite and conductor on Kubernetes. And i have the same question.

my conductor conf

# Servers.
conductor.grpc-server.enabled=false

# Database persistence type.
conductor.db.type=dynomite

# Dynomite Cluster details.
# format is host:port:rack separated by semicolon
conductor.redis.hosts=dynomite.conductor.svc.cluster.local:8102:us-east-1b

# Dynomite cluster name
conductor.redis.clusterName=dynomite

# Namespace for the keys stored in Dynomite/Redis
conductor.redis.workflowNamespacePrefix=conductor

# Namespace prefix for the dyno queues
conductor.redis.queueNamespacePrefix=conductor_queues

# No. of threads allocated to dyno-queues (optional)
queues.dynomite.threads=10

# By default with dynomite, we want the repairservice enabled
conductor.app.workflowRepairServiceEnabled=true

# Non-quorum port used to connect to local redis.  Used by dyno-queues.
# When using redis directly, set this to the same port as redis server
# For Dynomite, this is 22122 by default or the local redis-server port used by Dynomite.
conductor.redis.queuesNonQuorumPort=22122

# Elastic search instance indexing is enabled.
conductor.indexing.enabled=true

# Transport address to elasticsearch
conductor.elasticsearch.url=http://elasticsearch.conductor.svc.cluster.local:9200

# Name of the elasticsearch cluster
conductor.elasticsearch.indexName=conductor
#conductor.event-queues.amqp.queueType=classic
#conductor.event-queues.amqp.sequentialMsgProcessing=true

# Additional modules for metrics collection exposed via logger (optional)
# conductor.metrics-logger.enabled=true
# conductor.metrics-logger.reportPeriodSeconds=15

# Additional modules for metrics collection exposed to Prometheus (optional)
# conductor.metrics-prometheus.enabled=true
# management.endpoints.web.exposure.include=prometheus

# To enable Workflow/Task Summary Input/Output JSON Serialization, use the following:
# conductor.app.summary-input-output-json-serialization.enabled=true

# Load sample kitchen sink workflow
loadSample=true

And i use the default conf of dynomite.

this is the error information:

I checked the source code. It looks like something wired for build dependence. @ComponentScan(basePackages = {"com.netflix.conductor", "io.orkes.conductor"}) orkes played some tracks for that.

dynomite can't read lua script successfully/

Is there any idea of this?

[manan164]

Hi @lijia-rengage , Dynomite is no longer supported. Can you please try with latest build which has orkes-queue?

[lijia-rengage]

Hi @lijia-rengage , Dynomite is no longer supported. Can you please try with latest build which has orkes-queue?

Could you plz tell me why we should abandon the use of Dynomite? And what are the advantages of using Orkes-Queue instead? Thanks a lot! Probably I would use it in a prod environment, so I would pay attention to its performance and stability.

[Ash-win]

It is the same behavior with latest release.