search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
127 stars 1.04k forks source link

Setting KSQL_CONFIG_DIR can disable ksqlDB server logging #5839

Open mabouchard-estruxture opened 4 years ago

mabouchard-estruxture commented 4 years ago

Describe the bug We have a server on which kSQLdb 0.8.1 runs within a Docker container, under CentOS 7. Recently, we tried to upgrade to versions 0.9.0 and 0.10.1, but neither of these newer versions were functional.

On 0.9.0, the container started fine, but running the CLI gave the warning: WARNING: Unable to create a system terminal, creating a dumb terminal

On 0.10.1, the container fails to start at all, and gives the following messages:

Starting ksqldb-server ... done
Starting ksqldb-cli    ... done
Attaching to ksqldb-server, ksqldb-cli
ksqldb-server    | ===> Configuring ksqlDB...
ksqldb-server    | ===> Launching ksqlDB Server...
ksqldb-server    | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ksqldb-server    | SLF4J: A number (108) of logging calls during the initialization phase have been intercepted and are
ksqldb-server    | SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
ksqldb-server    | SLF4J: See also http://www.slf4j.org/codes.html#replay
ksqldb-server exited with code 255

To Reproduce

  1. Install docker: yum install docker
  2. Install docker-compose: sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  3. Apply the necessary permissions on the docker-compose folder: sudo chmod +x /usr/local/bin/docker-compose
  4. Create a docker-compose.yml file and fill it out as such: 
    
---
version: '2'

services: ksqldb-server: image: confluentinc/ksqldb-server:0.10.1 hostname: ksqldb-server container_name: ksqldb-server volumes:

Additional context We also attempted to install kSQLdb on another machine, also running CentOS 7. This time, we installed fresh. However, we encountered the same behavior, and ultimately installed 0.8.1 in order to have a functional kSQLdb to work with.

In both cases, kSQLdb successfully created its system topics in Kafka.

Both systems have the same docker system info, with only difference being the kernel's build number:

Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  WARNING: You're not using the default seccomp profile
  Profile: /etc/docker/seccomp.json
Kernel Version: 3.10.0-957.12.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 8
Total Memory: 62.76 GiB
Name: [OBFUSCATED]
ID: D4VY:THOA:Z7CR:X24E:CADG:NCQJ:MMPN:S6ZE:S26F:PAWJ:FPVW:GZQB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
colinhicks commented 4 years ago

We took a look at this in Community Slack. In 0.10.1, the config KSQL_CONFIG_DIR: "/etc/ksqldb" appears to disable logging. This masked a separate issue causing server startup to fail.

I'm going to update the issue title to focus on the configuration issue. In general the repro steps above apply.

vpapavas commented 4 years ago

I tried reproducing the error but I am unable to. I don't have CentOS so I tried on my mac with the following docker-compose file

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.5.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  broker:
    image: confluentinc/cp-enterprise-kafka:5.5.0
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.10.1
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksqldb"
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:9092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.10.1
    container_name: ksqldb-cli
    depends_on:
      - broker
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true
    environment:
      KSQL_CONFIG_DIR: "/etc/ksqldb"

The server started normally and I was able to see the logging