search

confluentinc / cp-docker-images

[DEPRECATED] Docker images for Confluent Platform.
Apache License 2.0
1.14k stars 704 forks source link

cp-kafka-rest in Kubernetes exits with error #286

Open rroopreddy opened 7 years ago

rroopreddy commented 7 years ago
  1. Starting Kafka Rest Proxy in Kubernetes exits with
   $ kubectl logs -f kafka-rest-lr991

   + [[ -n tcp://10.0.0.95:8082 ]]
  + echo 'PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.'
  + exit 1

  PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.

  1. Environment

  2. Followed this thread in Stackoverflow (https://stackoverflow.com/questions/41868161/kafka-in-kubernetes-cluster-how-to-publish-consume-messages-from-outside-of-kub/41869856).

  3. Following "kaka-rest-controller.yaml" file is used to start the pod in Kubernetes with required ENV variables

------------kaka-rest-controller.yaml-----------------
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    app: kafka-rest
    component: kafka-rest-endpoint
  name: kafka-rest
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka-rest
        component: kafka-rest-endpoint
    spec:
      containers:
      - name: kafka-rest
        image: confluentinc/cp-kafka-rest:3.2.2
        env:
          - name: KAFKA_REST_HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: KAFKA_REST_ZOOKEEPER_CONNECT
            value: zookeeper:2181

Not sure if there are additional environmental variables missing, By default the Docker image seem to set the PORT by default and seem to check for it to exit.

Any help is appreciated

rroopreddy commented 7 years ago 
This is what the compete startup log output looks like. there is a default env variable from docker image "KAFKA_REST_PORT=tcp://10.0.0.14:8082" that later gets checked to EXIT

$ kubectl logs -f kafka-rest-q0kwg
===> ENV Variables ...
APT_ALLOW_UNAUTHENTICATED=false
COMPONENT=kafka-rest
CONFLUENT_DEB_VERSION=2
CONFLUENT_MAJOR_VERSION=3
CONFLUENT_MINOR_VERSION=2
CONFLUENT_PATCH_VERSION=2
CONFLUENT_VERSION=3.2.2
HOME=/root
HOSTNAME=kafka-rest-q0kwg
KAFKA_ADVERTISED_HOST_NAME=172.17.0.9
KAFKA_PORT=tcp://10.0.0.233:9092
KAFKA_PORT_9092_TCP=tcp://10.0.0.233:9092
KAFKA_PORT_9092_TCP_ADDR=10.0.0.233
KAFKA_PORT_9092_TCP_PORT=9092
KAFKA_PORT_9092_TCP_PROTO=tcp
KAFKA_REST_HOST_NAME=172.17.0.9
KAFKA_REST_PORT=tcp://10.0.0.14:8082
KAFKA_REST_PORT_8082_TCP=tcp://10.0.0.14:8082
KAFKA_REST_PORT_8082_TCP_ADDR=10.0.0.14
KAFKA_REST_PORT_8082_TCP_PORT=8082
KAFKA_REST_PORT_8082_TCP_PROTO=tcp
KAFKA_REST_SERVICE_HOST=10.0.0.14
KAFKA_REST_SERVICE_PORT=8082
KAFKA_REST_SERVICE_PORT_KAFKA_REST_PORT=8082
KAFKA_REST_ZOOKEEPER_CONNECT=zookeeper:2181
KAFKA_SERVICE_HOST=10.0.0.233
KAFKA_SERVICE_PORT=9092
KAFKA_SERVICE_PORT_KAFKA_PORT=9092
KAFKA_VERSION=0.10.2.1
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
PYTHON_PIP_VERSION=8.1.2
PYTHON_VERSION=2.7.9-1
SCALA_VERSION=2.11
SHLVL=1
ZOOKEEPER_1_PORT=tcp://10.0.0.147:2181
ZOOKEEPER_1_PORT_2181_TCP=tcp://10.0.0.147:2181
ZOOKEEPER_1_PORT_2181_TCP_ADDR=10.0.0.147
ZOOKEEPER_1_PORT_2181_TCP_PORT=2181
ZOOKEEPER_1_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_1_PORT_2888_TCP=tcp://10.0.0.147:2888
ZOOKEEPER_1_PORT_2888_TCP_ADDR=10.0.0.147
ZOOKEEPER_1_PORT_2888_TCP_PORT=2888
ZOOKEEPER_1_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_1_PORT_3888_TCP=tcp://10.0.0.147:3888
ZOOKEEPER_1_PORT_3888_TCP_ADDR=10.0.0.147
ZOOKEEPER_1_PORT_3888_TCP_PORT=3888
ZOOKEEPER_1_PORT_3888_TCP_PROTO=tcp
ZOOKEEPER_1_SERVICE_HOST=10.0.0.147
ZOOKEEPER_1_SERVICE_PORT=2181
ZOOKEEPER_1_SERVICE_PORT_CLIENT=2181
ZOOKEEPER_1_SERVICE_PORT_ELECTION=3888
ZOOKEEPER_1_SERVICE_PORT_FOLLOWERS=2888
ZOOKEEPER_2_PORT=tcp://10.0.0.134:2181
ZOOKEEPER_2_PORT_2181_TCP=tcp://10.0.0.134:2181
ZOOKEEPER_2_PORT_2181_TCP_ADDR=10.0.0.134
ZOOKEEPER_2_PORT_2181_TCP_PORT=2181
ZOOKEEPER_2_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_2_PORT_2888_TCP=tcp://10.0.0.134:2888
ZOOKEEPER_2_PORT_2888_TCP_ADDR=10.0.0.134
ZOOKEEPER_2_PORT_2888_TCP_PORT=2888
ZOOKEEPER_2_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_2_PORT_3888_TCP=tcp://10.0.0.134:3888
ZOOKEEPER_2_PORT_3888_TCP_ADDR=10.0.0.134
ZOOKEEPER_2_PORT_3888_TCP_PORT=3888
ZOOKEEPER_2_PORT_3888_TCP_PROTO=tcp
ZOOKEEPER_2_SERVICE_HOST=10.0.0.134
ZOOKEEPER_2_SERVICE_PORT=2181
ZOOKEEPER_2_SERVICE_PORT_CLIENT=2181
ZOOKEEPER_2_SERVICE_PORT_ELECTION=3888
ZOOKEEPER_2_SERVICE_PORT_FOLLOWERS=2888
ZOOKEEPER_3_PORT=tcp://10.0.0.2:2181
ZOOKEEPER_3_PORT_2181_TCP=tcp://10.0.0.2:2181
ZOOKEEPER_3_PORT_2181_TCP_ADDR=10.0.0.2
ZOOKEEPER_3_PORT_2181_TCP_PORT=2181
ZOOKEEPER_3_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_3_PORT_2888_TCP=tcp://10.0.0.2:2888
ZOOKEEPER_3_PORT_2888_TCP_ADDR=10.0.0.2
ZOOKEEPER_3_PORT_2888_TCP_PORT=2888
ZOOKEEPER_3_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_3_PORT_3888_TCP=tcp://10.0.0.2:3888
ZOOKEEPER_3_PORT_3888_TCP_ADDR=10.0.0.2
ZOOKEEPER_3_PORT_3888_TCP_PORT=3888
ZOOKEEPER_3_PORT_3888_TCP_PROTO=tcp
ZOOKEEPER_3_SERVICE_HOST=10.0.0.2
ZOOKEEPER_3_SERVICE_PORT=2181
ZOOKEEPER_3_SERVICE_PORT_CLIENT=2181
ZOOKEEPER_3_SERVICE_PORT_ELECTION=3888
ZOOKEEPER_3_SERVICE_PORT_FOLLOWERS=2888
ZOOKEEPER_PORT=tcp://10.0.0.189:2181
ZOOKEEPER_PORT_2181_TCP=tcp://10.0.0.189:2181
ZOOKEEPER_PORT_2181_TCP_ADDR=10.0.0.189
ZOOKEEPER_PORT_2181_TCP_PORT=2181
ZOOKEEPER_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_SERVICE_HOST=10.0.0.189
ZOOKEEPER_SERVICE_PORT=2181
ZOOKEEPER_SERVICE_PORT_CLIENT=2181
ZULU_OPENJDK_VERSION=8=8.17.0.3
_=/usr/bin/env
===> User

. /etc/confluent/docker/apply-mesos-overrides
+ . /etc/confluent/docker/apply-mesos-overrides

#!/usr/bin/env bash
#
# Copyright 2016 Confluent Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Mesos DC/OS docker deployments will have HOST and PORT0
# set for the proxying of the service.
#
# Use those values provide things we know we'll need.

[ -n "${HOST:-}" ] && [ -z "${KAFKA_REST_HOST_NAME:-}" ] && \
    export KAFKA_REST_HOST_NAME=$HOST || true # we don't want the setup to fail if not on Mesos
++ '[' -n '' ']'
++ true

echo "===> ENV Variables ..."
+ echo '===> ENV Variables ...'
env | sort
+ env
+ sort

echo "===> User"
+ echo '===> User'
id
+ id

echo "===> Configuring ..."
+ echo '===> Configuring ...'
/etc/confluent/docker/configure
+ /etc/confluent/docker/configure
uid=0(root) gid=0(root) groups=0(root)
===> Configuring ...

dub ensure KAFKA_REST_ZOOKEEPER_CONNECT
+ dub ensure KAFKA_REST_ZOOKEEPER_CONNECT
dub ensure KAFKA_REST_HOST_NAME
+ dub ensure KAFKA_REST_HOST_NAME

dub path /etc/"${COMPONENT}"/ writable
+ dub path /etc/kafka-rest/ writable

if [[ -n "${KAFKA_REST_PORT-}" ]]
then
  echo "PORT is deprecated. Please use KAFKA_REST_LISTENERS instead."
  exit 1
fi
+ [[ -n tcp://10.0.0.14:8082 ]]
+ echo 'PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.'
+ exit 1
PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.
geofflamrock commented 7 years ago

Do you have a Kubernetes service called kafka-rest in your setup?

Environment variables are added by Kubernetes for services by default, and I found that I had to name the service something else like kafka-rest-service for it to not find that environment variable and continue on.

bojidar-bg commented 5 years ago

I managed to workaround this issue without renaming the service by unsetting the environment variable before running:

apiVersion: apps/v1
kind: StatefulSet
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: confluentinc/cp-kafka-rest:5.0.1
        # ...
        command: ["bash", "-c", "unset KAFKA_REST_PORT; /etc/confluent/docker/run"]
Aidan-Pasquale99 commented 4 years ago

I managed to workaround this issue without renaming the service by unsetting the environment variable before running:

apiVersion: apps/v1
kind: StatefulSet
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: confluentinc/cp-kafka-rest:5.0.1
        # ...
        command: ["bash", "-c", "unset KAFKA_REST_PORT; /etc/confluent/docker/run"]

This fixed the same issue when deploying the Schema Registry image, thank you!

tirelibirefe commented 4 years ago

I'm trying to install schema-registry via Helm helm install incubator/schema-registry --version 1.1.9 The problem is still exist since 2017. Could you please give more details about the workaround the above? Where/How can I apply that workaround in helm charts?

b4nnertail commented 4 years ago

I'm trying to install schema-registry via Helm helm install incubator/schema-registry --version 1.1.9 The problem is still exist since 2017. Could you please give more details about the workaround the above? Where/How can I apply that workaround in helm charts?

I am also using incubator/schema-registry and could resolve the issue by simply changing the release name from schema-registry to anything else, in this case sr:

helm install sr -f schema-registry_values.yaml incubator/schema-registry

tirelibirefe commented 4 years ago

I'm trying to install schema-registry via Helm helm install incubator/schema-registry --version 1.1.9 The problem is still exist since 2017. Could you please give more details about the workaround the above? Where/How can I apply that workaround in helm charts?

I am also using incubator/schema-registry and could resolve the issue by simply changing the release name from schema-registry to anything else, in this case sr:

helm install sr -f schema-registry_values.yaml incubator/schema-registry

It was about almost 2 months ago. Currently I have a working system but I really don't remember anything about workaround.

WZ commented 4 years ago

This is because, by default, kubelet injects a set of ENVs automatically (for service discovery purpose). Another workaround is to disable this by setting enableServiceLinks: false in the pod spec (https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#accessing-the-service)

raohammad commented 3 years ago

I managed to workaround this issue without renaming the service by unsetting the environment variable before running:

apiVersion: apps/v1
kind: StatefulSet
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: confluentinc/cp-kafka-rest:5.0.1
        # ...
        command: ["bash", "-c", "unset KAFKA_REST_PORT; /etc/confluent/docker/run"]

This approach worked for me. Just that in my case, I did not have KAFKA_REST_PORT rather KAFKA_PORT. Finally this is how the command looks like for my deployment template of Kafka

command: ["bash", "-c", "unset KAFKA_PORT; /etc/confluent/docker/run"]

Tarasovych commented 2 years ago

Same issue with cp-kafka (0.6.1), but there is no way to override command