confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
127 stars 1.04k forks source link

KSqlDb REST Api returns http response 0 #7669

Closed MarcPhished closed 3 years ago

MarcPhished commented 3 years ago

Describe the bug I call the ksql API to create a stream and it returns 0. This only happens when my client is running in Docker. Posted this on Slack and was asked to open an issue here.

"Michael Drogalis Can you open a GitHub issue and post the full set of steps to reproduce? In particular, if you could show us more of the response, that'd help us best figure out what's up."

To Reproduce

Docker file

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: Zookeeper
    networks: 
      - broker-kafka
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: Kafka
    networks: 
      - broker-kafka
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_CLEANER_DELETE_RETENTION_MS: 5000
      KAFKA_BROKER_ID: 1
      KAFKA_MIN_INSYNC_REPLICAS: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_FFSETS_TOPIC_REPLICATION_FACTOR: 1

ksqldb-server:
        image: confluentinc/ksqldb-server:0.17.0
        hostname: ksqldb-server
        container_name: KsqlDb
        networks: 
          - broker-kafka
        depends_on:
          - kafka
          - schema-registry
        ports:
          - "8088:8088"
        environment:
          KSQL_LISTENERS: http://0.0.0.0:8088
          KSQL_BOOTSTRAP_SERVERS: kafka:29092
          KSQL_KSQL_SCHEMA_REGISTRY_URL: http://schema-registry:8081
          KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
          KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
          KSQL_KSQL_QUERY_PULL_TABLE_SCAN_ENABLED: "true"

  ksqldb-cli:
        image: confluentinc/ksqldb-cli
        container_name: KsqlDbCli
        networks: 
          - broker-kafka
        depends_on:
           - zookeeper
           - ksqldb-server
        entrypoint: /bin/sh
        tty: true 

licence-poller:  (my client application)
    build:
      context: .
      dockerfile: ./Dockerfiles/dockerfile.Licence.Poller
    container_name: PhishedLicencePoller
    environment:
      - ASPNETCORE_ENVIRONMENT=Docker
      - Kafka__Producers__BootstrapServers=kafka:29092
      - KSqlDb__Url=ksqldb-server:8088
    depends_on:
      - kafka
      - mongodb
    networks: 
      - broker-kafka
    restart: on-failure

volumes:
  mongodb_data:

networks: 
  broker-kafka:
    driver: bridge  

My client licence-poller calling the ksql endpoint

        public void CreateStreamFromPayload<TPayload>(string streamName, string topicName)
        {
            Console.WriteLine($"Creating KSqlDb stream {streamName}");

            ..

            var queryUrl = _options.Value.Url.TrimEnd('/') + "/ksql"; //read from KSqlDb.Url  -> KSqlDb__Url=ksqldb-server:8088

            var client = new RestClient(queryUrl)
            {
                Timeout = 3000
            };

            var script = CreateStreamScriptFromPayload<TPayload>(streamName, topicName);

            var request = new RestRequest(Method.POST);
            request.AddHeader("Content-Type", "application/json");
            request.AddParameter("application/json",
                $@"{{
                      ""ksql"": ""{script}"",
                        ""streamsProperties"": {{
                           ""auto.offset.reset"": ""earliest""
                      }}
                    }}",
                ParameterType.RequestBody);

            try
            {
                client.Execute(request);
            }
            catch (Exception e)
            {
                Console.WriteLine($"Error created KSqlDb stream {streamName} {e.Message}");
            }
        }

Expected behavior The stream is created and 200 is returned.

Actual behaviour Request returns HTTP status code 0. Nothing appears in the KSqlDb logs.

Additional context When I run my client licence-poller outside it's container, everything works fine.

MarcPhished commented 3 years ago

Issue is solved by using http://ksqldb-server:8088