ksqlDB exited with code 255 after restart the docker compose

[samyujialiu]

HI there: We are running our kafka on docker compose. It works fine if Kafka starts the first time, while KSQLDB crashed if we restart it. Now, the only solution is delete the persistent files (zookeeper and broker) and restart them. Can you pls help us find the problem? Thanks a lot The docker compose are below:

version: '3.3' services: zookeeper: image: confluentinc/cp-zookeeper:${CONFLUENT_VERSION} hostname: zookeeper ports:

• "${ZOOKEEPER_EXTERNAL_PORT}:${ZOOKEEPER_INTERNAL_PORT}" volumes:
• "${PERSISTENCE_ROOT}/zk-data:/var/lib/zookeeper/data"
• "${PERSISTENCE_ROOT}/zk-txn-logs:/var/lib/zookeeper/log" environment: ZOOKEEPER_CLIENT_PORT: ${ZOOKEEPER_INTERNAL_PORT} ZOOKEEPER_TICK_TIME: 2000 env_file:

• ./noproxy.conf

  broker: image: confluentinc/cp-enterprise-kafka:${CONFLUENT_VERSION} hostname: broker depends_on:

• zookeeper ports:
• "${BROKER_LISTENER_EXTERNAL_PORT}:${BROKER_LISTENER_INTERNAL_PORT}"
• "${BROKER_BOOTSTRAP_EXTERNAL_PORT}:${BROKER_BOOTSTRAP_INTERNAL_PORT}" volumes:
• "${PERSISTENCE_ROOT}/kafka-data:/var/lib/kafka/data" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:${ZOOKEEPER_INTERNAL_PORT} KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:${BROKER_LISTENER_INTERNAL_PORT},PLAINTEXT_HOST://localhost:${BROKER_BOOTSTRAP_INTERNAL_PORT} CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:${BROKER_LISTENER_INTERNAL_PORT} CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:${ZOOKEEPER_INTERNAL_PORT} KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100 CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:${BROKER_LISTENER_INTERNAL_PORT} CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:${ZOOKEEPER_INTERNAL_PORT} CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1 CONFLUENT_METRICS_ENABLE: 'TRUE' ####### env_file:
• ./proxy.conf
• ./noproxy.conf schema-registry: image: confluentinc/cp-schema-registry:${CONFLUENT_VERSION} hostname: schema-registry depends_on:
• zookeeper
• broker ports:
• "${SCHEMA_REGISTRY_EXTERNAL_PORT}:${SCHEMA_REGISTRY_INTERNAL_PORT}" volumes:
• "./kafka/schema-registry.properties.template:/etc/confluent/docker/schema-registry.properties.template" environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:${ZOOKEEPER_INTERNAL_PORT}' env_file:
• ./noproxy.conf schema-registry-ui: image: landoop/schema-registry-ui:${SCHEMA_REGISTRY_UI_VERSION} hostname: schema-registry-ui depends_on:
• schema-registry ports:
• ${SCHEMA_REGISTRY_UI_EXTERNAL_PORT}:${SCHEMA_REGISTRY_UI_INTERNAL_PORT} environment: SCHEMAREGISTRY_URL: http://127.0.0.1:${SCHEMA_REGISTRY_EXTERNAL_PORT} env_file:
• ./noproxy.conf connect: build: context: . dockerfile: kafka/Dockerfile hostname: connect depends_on:
• zookeeper
• broker
• schema-registry ports:
• "${CONNECT_EXTERNAL_PORT}:${CONNECT_INTERNAL_PORT}" volumes:

  A 'safer' host mount. Using ${HOME} can be very dangerous

• ./host-safe/connect:/data/shared/safe

  Overwrite the included Debezium DB2 connector JARS

  - ./host-safe/debezium-db2-jars:/usr/share/java/kafka/debezium-connector-db2

  environment: CONNECT_BOOTSTRAP_SERVERS: 'broker:${BROKER_LISTENER_INTERNAL_PORT}' CONNECT_REST_ADVERTISED_HOST_NAME: connect CONNECT_REST_PORT: ${CONNECT_INTERNAL_PORT} CONNECT_GROUP_ID: compose-connect-group CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1 CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000 CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1 CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1 CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:${SCHEMA_REGISTRY_INTERNAL_PORT} CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter" CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter" CONNECT_ZOOKEEPER_CONNECT: 'zookeeper:${ZOOKEEPER_INTERNAL_PORT}'

  Assumes image is based on confluentinc/kafka-connect-datagen:${CONFLUENT_VERSION} which is pulling 5.1.1 Connect image

  CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-${CONFLUENT_VERSION}.jar CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor" CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"

  Include a volume mounted plugins path for testing plugins

  CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components,/etc/kafka-connect/jars,/data/shared/safe/connect-plugins" CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY: 'All' command: "bash -c 'if [ ! -d /usr/share/confluent-hub-components/confluentinc-kafka-connect-datagen ]; then echo \"WARNING: Did not find directory for kafka-connect-datagen (did you remember to run: docker-compose up -d --build ?)\"; fi ; /etc/confluent/docker/run'" env_file:

• ./proxy.conf
• ./noproxy.conf control-center: image: confluentinc/cp-enterprise-control-center:${CONFLUENT_VERSION} hostname: control-center depends_on:
• zookeeper
• broker
• schema-registry
• connect
• ksql-server ports:
• "${CONTROL_CENTER_EXTERNAL_PORT}:${CONTROL_CENTER_INTERNAL_PORT}" environment: CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:${BROKER_LISTENER_INTERNAL_PORT}' CONTROL_CENTER_ZOOKEEPER_CONNECT: 'zookeeper:${ZOOKEEPER_INTERNAL_PORT}' CONTROL_CENTER_CONNECT_CLUSTER: 'connect:${CONNECT_INTERNAL_PORT}' CONTROL_CENTER_KSQL_URL: "http://ksql-server:${KSQL_SERVER_INTERNAL_PORT}" CONTROL_CENTER_KSQL_ADVERTISED_URL: "http://localhost:${KSQL_SERVER_INTERNAL_PORT}" CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:${SCHEMA_REGISTRY_INTERNAL_PORT}" CONTROL_CENTER_REPLICATION_FACTOR: 1 CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1 CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1 CONFLUENT_METRICS_TOPIC_REPLICATION: 1 PORT: ${CONTROL_CENTER_INTERNAL_PORT} env_file:
• ./proxy.conf
• ./noproxy.conf ksql-server: image: confluentinc/ksqldb-server:${CONFLUENT_VERSION} hostname: ksql-server depends_on:
• broker
• schema-registry ports:
• "${KSQL_SERVER_EXTERNAL_PORT}:${KSQL_SERVER_INTERNAL_PORT}" environment: KSQL_LISTENERS: "http://0.0.0.0:${KSQL_SERVER_EXTERNAL_PORT}" KSQL_BOOTSTRAP_SERVERS: 'broker:${BROKER_LISTENER_INTERNAL_PORT}' KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:${SCHEMA_REGISTRY_INTERNAL_PORT} KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true" KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true" KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:${SCHEMA_REGISTRY_INTERNAL_PORT}" env_file:

• ./noproxy.conf

  ksql-cli: image: confluentinc/ksqldb-cli:${CONFLUENT_VERSION} hostname: ksql-cli depends_on:

• ksql-server entrypoint: /bin/sh tty: true env_file:
• ./noproxy.conf

  ksql-datagen:

  Downrev ksql-examples to 5.1.2 due to DEVX-798 (work around issues in 5.2.0)

  image: confluentinc/ksql-examples:${CONFLUENT_VERSION}

  hostname: ksql-datagen

  depends_on:

  - ksql-server

  - broker

  - schema-registry

  - connect

  command: "bash -c 'echo Waiting for Kafka to be ready... && \

                 #cub kafka-ready -b broker:${BROKER_LISTENER_INTERNAL_PORT} 1 40 && \
                 #echo Waiting for Confluent Schema Registry to be ready... && \
                 #cub sr-ready schema-registry ${SCHEMA_REGISTRY_INTERNAL_PORT} 40 && \
                 #echo Waiting a few seconds for topic creation to finish... && \
                 #sleep 11 && \
                 #tail -f /dev/null'"

  environment:

  KSQL_CONFIG_DIR: "/etc/ksql"

  KSQL_LOG4J_OPTS: "-Dlog4j.configuration=file:/etc/ksql/log4j-rolling.properties"

  STREAMS_BOOTSTRAP_SERVERS: broker:${BROKER_LISTENER_INTERNAL_PORT}

  STREAMS_SCHEMA_REGISTRY_HOST: schema-registry

  STREAMS_SCHEMA_REGISTRY_PORT: ${SCHEMA_REGISTRY_INTERNAL_PORT}

  env_file:

  - ./noproxy.conf

  rest-proxy: image: confluentinc/cp-kafka-rest:${CONFLUENT_VERSION} hostname: rest-proxy depends_on:

• zookeeper
• broker
• schema-registry ports:
• ${REST_PROXY_EXTERNAL_PORT}:${REST_PROXY_INTERNAL_PORT} environment: KAFKA_REST_HOST_NAME: rest-proxy KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:${BROKER_LISTENER_INTERNAL_PORT}' KAFKA_REST_LISTENERS: "http://0.0.0.0:${REST_PROXY_INTERNAL_PORT}" KAFKA_REST_SCHEMA_REGISTRY_URL: 'http://schema-registry:${SCHEMA_REGISTRY_INTERNAL_PORT}' env_file:
• ./noproxy.conf

[big-andy-coates]

Hi @samyujialiu,

Can I ask what you've done to investigate the issue so far?

The error message is saying that the command topic (with a name I can't copy here as its in an image), doesn't exist.

Have you confirmed it does or doesn't exist in the Kafka cluster?

If it does exist, it could be an ACLs problem if you have them enabled. Does the ksqlDB user have permissions to describe the cluster and/or topic?