search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
41 stars 1.04k forks source link

CREATE/SHOW CONNECTOR Failure #4127

Closed Sarlken closed 4 years ago

Sarlken commented 4 years ago

ksql> SHOW CONNECTORS;io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8083 [localhost/127.0.0.1] failed: Connection refused (Connection refused) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8083 [localhost/127.0.0.1] failed: Connection refused (Connection refused) Caused by: Could not connect to the server. Caused by: Could not connect to the server. After installing KSQLDB to my virtual machine, when I try to create JDBC Connector for my mysql db I receive the error below:

io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8083 [localhost/127.0.0.1] failed: Connection refused (Connection refused) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8083 [localhost/127.0.0.1] failed: Connection refused (Connection refused) Caused by: Could not connect to the server. Caused by: Could not connect to the server.

I receive the same error when I try to create the connector. I think it's related with Kafka Connect component inside of the KSQLDB, since all the queries related with the KSQL part is functioning really well. However, all the queries related with Kafka Connector fails and gives the same error message as above.

The versions are CLI v0.6.0, Server v0.6.0 The SQL statements I run SHOW CONNECTORS;

I would be glad for any help about how to resolve this error.

agavra commented 4 years ago

The reason this doesn't work is that you need to configure ksqlDB with a connect property file.

If you are intending to use a local embedded connect worker, you need to supply a connect.properties file to ksqlDB. You can take a look at https://github.com/confluentinc/ksql/blob/master/config/connect.properties for a sample configuration (make sure to update the bootstrap.server and the schema registry URLs). You will need to mount this file via docker volumes (we're working on a way to make this easier) and then set KSQL_KSQL_CONNECT_WORKER_CONFIG: /path/to/volume/connect.properties

Sarlken commented 4 years ago

Thanks for your response. I modified connect.properties and created a path in my virtual machine as you said, then copied the file to the path and added the line below to the ksqldb's docker-compose.yml file as below:

**version: "3"
services:
    schema-registry:
    image: "[my_image_path]"
    networks:
      ntwrk-zkpr-stck:
      ntwrk-kfk-stck:
    depends_on:
      - srvc-apch-zkpr0
      - srvc-apch-zkpr1
      - srvc-apch-zkpr2
      - srvc-kfk-brkr0
      - srvc-kfk-brkr1
      - srvc-kfk-brkr2
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: srvc-apch-zkpr0:2181, srvc-apch-zkpr0:2181, srvc-apch-zkpr0:2181
      SCHEMA_REGISTRY_LISTENERS: "http://0.0.0.0:8081"
    ports:
      - 8081:8081
  ksqldb-server:
    image: [my_image_path]
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8089:8089"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8089
      KSQL_BOOTSTRAP_SERVERS: [ip_of_my_vm]:9092,[ip_of_my_vm]:9093,[ip_of_my_vm]:9094
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_CONNECT_WORKER_CONFIG: /etc/ksqldb/config/connect.properties
  ksqldb-cli:
    image: [my_image_path]
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true
networks:
  ntwrk-zkpr-stck:
  ntwrk-kfk-stck:**

PS: Image values are the docker repositories that I pushed so no problem about them.

After doing these configurations, ksqldb-server is always failing. I couldn't understand why.

In this point, I would like to ask whether kSQLDB has an embedded schema registry or do I need to install schema registry in the docker-compose file above?

Thanks for the help.

McHardex commented 4 years ago

Hi @agavra , I tried the solution you suggested above and after doing so, I keep getting the same error just like @Sarlken mentioned.

Here is my docker-compose.yml file

version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-enterprise-kafka:5.3.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

  schema-registry:
    image: confluentinc/cp-schema-registry:5.3.1
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181

  connect:
    image: confluentinc/cp-kafka-connect:latest
    hostname: connect
    container_name: connect
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: broker:29092
      CONNECT_REST_ADVERTISED_HOST_NAME: localhost
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: ksql-connect-cluster
      CONNECT_OFFSET_STORAGE_TOPIC: ksql-connect-configs
      CONNECT_CONFIG_STORAGE_TOPIC: ksql-connect-topics
      CONNECT_STATUS_STORAGE_TOPIC: ksql-connect-statuses
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      # CLASSPATH required due to CC-2422
      CLASSPATH: C:/Users/User01/Documents/confluent/share/confluent-hub-components
      # CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      # CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      CONNECT_PLUGIN_PATH: C:/Users/User01/Documents/confluent/share/confluent-hub-components
      # CONNECT_LOG4J_LOGGERS: org.apache.zookeeper=ERROR,org.I0Itec.zkclient=ERROR,org.reflections=ERROR
      KSQL_KSQL_CONNECT_WORKER_CONFIG: C:/Users/User01/Documents/confluent/share/java/connect.properties
    volumes: 
      - C:/Users/User01/Documents/confluent/share/java/connect.properties

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.6.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_CONNECT_WORKER_CONFIG: C:/Users/User01/Documents/confluent/share/java/connect.properties
    volumes: 
      - C:/Users/User01/Documents/confluent/share/java/connect.properties

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.6.0
    container_name: ksqldb-cli
    depends_on:
      - broker
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

  postgres:
    # *-----------------------------*
    # To connect to the DB: 
    #   docker-compose exec postgres bash -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
    # *-----------------------------*
    image: postgres
    hostname: postgres
    container_name: postgres
    environment:
     - POSTGRES_USER=postgres
     - POSTGRES_PASSWORD=postgres
    # volumes:
    #  - ./data/postgres:/docker-entrypoint-initdb.d/

I checked the log of the ksqldb-server container and this is the response:


===> Configuring ksqlDB...
===> Launching ksqlDB Server...
[2019-12-23 10:59:56,473] INFO KsqlConfig values:
        ksql.access.validator.enable = auto
        ksql.avro.maps.named = true
        ksql.connect.url = http://localhost:8083
        ksql.connect.worker.config = C:/Users/User01/Documents/confluent/share/java/connect.properties
        ksql.extension.dir = ext
        ksql.functions.substring.legacy.args = false
        ksql.insert.into.values.enabled = true
        ksql.internal.topic.replicas = 1
        ksql.metric.reporters = []
        ksql.metrics.extension = null
        ksql.metrics.tags.custom =
        ksql.named.internal.topics = on
        ksql.output.topic.name.prefix =
        ksql.persistence.wrap.single.values = true
        ksql.persistent.prefix = query_
        ksql.query.inject.legacy.map.values.node = false
        ksql.query.persistent.active.limit = 2147483647
        ksql.query.pull.enable = true
        ksql.query.pull.routing.timeout.ms = 30000
        ksql.query.pull.skip.access.validator = false
        ksql.query.pull.streamsstore.rebalancing.timeout.ms = 10000
        ksql.query.stream.groupby.rowkey.repartition = false
        ksql.schema.registry.url = http://localhost:8081
        ksql.security.extension.class = null
        ksql.service.id = default_
        ksql.sink.partitions = null
        ksql.sink.replicas = null
        ksql.sink.window.change.log.additional.retention = 1000000
        ksql.transient.prefix = transient_
        ksql.udf.collect.metrics = false
        ksql.udf.enable.security.manager = true
        ksql.udfs.enabled = true
        ksql.windowed.session.key.legacy = false
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
 (io.confluent.ksql.util.KsqlConfig:347)
[2019-12-23 10:59:56,563] INFO KsqlRestConfig values:
        access.control.allow.headers =
        access.control.allow.methods =
        access.control.allow.origin =
        authentication.method = NONE
        authentication.realm =
        authentication.roles = [*]
        authentication.skip.paths = []
        compression.enable = true
        debug = false
        idle.timeout.ms = 30000
        ksql.healthcheck.interval.ms = 5000
        ksql.server.command.response.timeout.ms = 5000
        ksql.server.exception.uncaught.handler.enable = false
        ksql.server.install.dir = /usr
        ksql.server.preconditions = []
        ksql.server.websockets.num.threads = 5
        listeners = [http://0.0.0.0:8088]
        metric.reporters = []
        metrics.jmx.prefix = rest-utils
        metrics.num.samples = 2
        metrics.sample.window.ms = 30000
        metrics.tag.map = []
        port = 8080
        query.stream.disconnect.check = 1000
        request.logger.name = io.confluent.rest-utils.requests
        resource.extension.classes = []
        response.mediatype.default = application/json
        response.mediatype.preferred = [application/json]
        rest.servlet.initializor.classes = []
        shutdown.graceful.ms = 1000
        ssl.cipher.suites = []
        ssl.client.auth = false
        ssl.client.authentication = NONE
        ssl.enabled.protocols = []
        ssl.endpoint.identification.algorithm = null
        ssl.key.password = [hidden]
        ssl.keymanager.algorithm =
        ssl.keystore.location =
        ssl.keystore.password = [hidden]
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider =
        ssl.trustmanager.algorithm =
        ssl.truststore.location =
        ssl.truststore.password = [hidden]
        ssl.truststore.type = JKS
        websocket.path.prefix = /ws
        websocket.servlet.initializor.classes = []
 (io.confluent.ksql.rest.server.KsqlRestConfig:347)
[2019-12-23 10:59:56,630] INFO Logging initialized @2909ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:169)
[2019-12-23 10:59:56,950] INFO Adding listener: http://0.0.0.0:8088 (io.confluent.rest.ApplicationServer:314)
[2019-12-23 10:59:57,055] INFO KsqlConfig values:
        ksql.access.validator.enable = auto
        ksql.avro.maps.named = true
        ksql.connect.url = http://localhost:8083
        ksql.connect.worker.config = C:/Users/User01/Documents/confluent/share/java/connect.properties
        ksql.extension.dir = ext
        ksql.functions.substring.legacy.args = false
        ksql.insert.into.values.enabled = true
        ksql.internal.topic.replicas = 1
        ksql.metric.reporters = []
        ksql.metrics.extension = null
        ksql.metrics.tags.custom =
        ksql.named.internal.topics = on
        ksql.output.topic.name.prefix =
        ksql.persistence.wrap.single.values = true
        ksql.persistent.prefix = query_
        ksql.query.inject.legacy.map.values.node = false
        ksql.query.persistent.active.limit = 2147483647
        ksql.query.pull.enable = true
        ksql.query.pull.routing.timeout.ms = 30000
        ksql.query.pull.skip.access.validator = false
        ksql.query.pull.streamsstore.rebalancing.timeout.ms = 10000
        ksql.query.stream.groupby.rowkey.repartition = false
        ksql.schema.registry.url = http://localhost:8081
        ksql.security.extension.class = null
        ksql.service.id = default_
        ksql.sink.partitions = null
        ksql.sink.replicas = null
        ksql.sink.window.change.log.additional.retention = 1000000
        ksql.transient.prefix = transient_
        ksql.udf.collect.metrics = false
        ksql.udf.enable.security.manager = true
        ksql.udfs.enabled = true
        ksql.windowed.session.key.legacy = false
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
 (io.confluent.ksql.util.KsqlConfig:347)
[2019-12-23 10:59:57,187] INFO KsqlConfig values:
        ksql.access.validator.enable = auto
        ksql.avro.maps.named = true
        ksql.connect.url = http://localhost:8083
        ksql.connect.worker.config = C:/Users/User01/Documents/confluent/share/java/connect.properties
        ksql.extension.dir = ext
        ksql.functions.substring.legacy.args = false
        ksql.insert.into.values.enabled = true
        ksql.internal.topic.replicas = 1
        ksql.metric.reporters = []
        ksql.metrics.extension = null
        ksql.metrics.tags.custom =
        ksql.named.internal.topics = on
        ksql.output.topic.name.prefix =
        ksql.persistence.wrap.single.values = true
        ksql.persistent.prefix = query_
        ksql.query.inject.legacy.map.values.node = false
        ksql.query.persistent.active.limit = 2147483647
        ksql.query.pull.enable = true
        ksql.query.pull.routing.timeout.ms = 30000
        ksql.query.pull.skip.access.validator = false
        ksql.query.pull.streamsstore.rebalancing.timeout.ms = 10000
        ksql.query.stream.groupby.rowkey.repartition = false
        ksql.schema.registry.url = http://localhost:8081
        ksql.security.extension.class = null
        ksql.service.id = default_
        ksql.sink.partitions = null
        ksql.sink.replicas = null
        ksql.sink.window.change.log.additional.retention = 1000000
        ksql.transient.prefix = transient_
        ksql.udf.collect.metrics = false
        ksql.udf.enable.security.manager = true
        ksql.udfs.enabled = true
        ksql.windowed.session.key.legacy = false
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.endpoint.identification.algorithm = https
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
 (io.confluent.ksql.util.KsqlConfig:347)
[2019-12-23 10:59:57,262] INFO ProcessingLogConfig values:
        ksql.logging.processing.rows.include = false
        ksql.logging.processing.stream.auto.create = true
        ksql.logging.processing.stream.name = KSQL_PROCESSING_LOG
        ksql.logging.processing.topic.auto.create = true
        ksql.logging.processing.topic.name =
        ksql.logging.processing.topic.partitions = 1
        ksql.logging.processing.topic.replication.factor = 1
 (io.confluent.ksql.logging.processing.ProcessingLogConfig:347)
[2019-12-23 10:59:58,535] INFO Blacklist file: /usr/ext/resource-blacklist.txt not found. No classes will be blacklisted (io.confluent.ksql.function.Blacklist:55)
[2019-12-23 10:59:59,225] INFO Adding function ENTRIES for method public java.util.List io.confluent.ksql.function.udf.array.Entries.entriesInt(java.util.Map,boolean) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,709] INFO Adding function ENTRIES for method public java.util.List io.confluent.ksql.function.udf.array.Entries.entriesBigInt(java.util.Map,boolean) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,718] INFO Adding function ENTRIES for method public java.util.List io.confluent.ksql.function.udf.array.Entries.entriesDouble(java.util.Map,boolean) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,738] INFO Adding function ENTRIES for method public java.util.List io.confluent.ksql.function.udf.array.Entries.entriesBoolean(java.util.Map,boolean) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,749] INFO Adding function ENTRIES for method public java.util.List io.confluent.ksql.function.udf.array.Entries.entriesString(java.util.Map,boolean) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,767] INFO Adding function GENERATE_SERIES for method public java.util.List io.confluent.ksql.function.udf.array.GenerateSeries.generateSeriesInt(int,int,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,771] INFO Adding function GENERATE_SERIES for method public java.util.List io.confluent.ksql.function.udf.array.GenerateSeries.generateSeriesInt(int,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,784] INFO Adding function GENERATE_SERIES for method public java.util.List io.confluent.ksql.function.udf.array.GenerateSeries.generateSeriesLong(long,long,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,789] INFO Adding function GENERATE_SERIES for method public java.util.List io.confluent.ksql.function.udf.array.GenerateSeries.generateSeriesLong(long,long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,868] INFO Adding function datetostring for method public java.lang.String io.confluent.ksql.function.udf.datetime.DateToString.dateToString(int,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,871] INFO Adding function stringtodate for method public int io.confluent.ksql.function.udf.datetime.StringToDate.stringToDate(java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,898] INFO Adding function stringtotimestamp for method public long io.confluent.ksql.function.udf.datetime.StringToTimestamp.stringToTimestamp(java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,899] INFO Adding function stringtotimestamp for method public long io.confluent.ksql.function.udf.datetime.StringToTimestamp.stringToTimestamp(java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,908] INFO Adding function timestamptostring for method public java.lang.String io.confluent.ksql.function.udf.datetime.TimestampToString.timestampToString(long,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,908] INFO Adding function timestamptostring for method public java.lang.String io.confluent.ksql.function.udf.datetime.TimestampToString.timestampToString(long,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,910] INFO Adding function unix_date for method public int io.confluent.ksql.function.udf.datetime.UnixDate.unixDate()
(io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,911] INFO Adding function unix_timestamp for method public long io.confluent.ksql.function.udf.datetime.UnixTimestamp.unixTimestamp() (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,915] INFO Adding function geo_distance for method public java.lang.Double io.confluent.ksql.function.udf.geo.GeoDistance.geoDistance(double,double,double,double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,916] INFO Adding function geo_distance for method public java.lang.Double io.confluent.ksql.function.udf.geo.GeoDistance.geoDistance(double,double,double,double,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,948] INFO Adding function AS_ARRAY for method public final java.util.List io.confluent.ksql.function.udf.list.AsArray.asArray(java.lang.Object[]) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,950] INFO Adding function slice for method public java.util.List io.confluent.ksql.function.udf.list.Slice.slice(java.util.List,java.lang.Integer,java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,951] INFO Adding function AS_MAP for method public final java.util.Map io.confluent.ksql.function.udf.map.AsMap.asMap(java.util.List,java.util.List) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,954] INFO Adding function Abs for method public java.lang.Double io.confluent.ksql.function.udf.math.Abs.abs(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,955] INFO Adding function Abs for method public java.math.BigDecimal io.confluent.ksql.function.udf.math.Abs.abs(java.math.BigDecimal) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,956] INFO Adding function Abs for method public java.lang.Double io.confluent.ksql.function.udf.math.Abs.abs(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,957] INFO Adding function Abs for method public java.lang.Double io.confluent.ksql.function.udf.math.Abs.abs(java.lang.Long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,997] INFO Adding function exp for method public java.lang.Double io.confluent.ksql.function.udf.math.Exp.exp(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 10:59:59,999] INFO Adding function exp for method public java.lang.Double io.confluent.ksql.function.udf.math.Exp.exp(java.lang.Long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,000] INFO Adding function exp for method public java.lang.Double io.confluent.ksql.function.udf.math.Exp.exp(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,002] INFO Adding function Floor for method public java.lang.Double io.confluent.ksql.function.udf.math.Floor.floor(java.math.BigDecimal) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,004] INFO Adding function Floor for method public java.lang.Double io.confluent.ksql.function.udf.math.Floor.floor(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,006] INFO Adding function Floor for method public java.lang.Double io.confluent.ksql.function.udf.math.Floor.floor(java.lang.Long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,027] INFO Adding function Floor for method public java.lang.Double io.confluent.ksql.function.udf.math.Floor.floor(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,030] INFO Adding function ln for method public java.lang.Double io.confluent.ksql.function.udf.math.Ln.ln(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,030] INFO Adding function ln for method public java.lang.Double io.confluent.ksql.function.udf.math.Ln.ln(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,031] INFO Adding function ln for method public java.lang.Double io.confluent.ksql.function.udf.math.Ln.ln(java.lang.Long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,039] INFO Adding function Round for method public java.math.BigDecimal io.confluent.ksql.function.udf.math.Round.round(java.math.BigDecimal) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,039] INFO Adding function Round for method public java.lang.Long io.confluent.ksql.function.udf.math.Round.round(long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,040] INFO Adding function Round for method public java.lang.Long io.confluent.ksql.function.udf.math.Round.round(int)
(io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,048] INFO Adding function Round for method public java.math.BigDecimal io.confluent.ksql.function.udf.math.Round.round(java.math.BigDecimal,java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,068] INFO Adding function Round for method public java.lang.Long io.confluent.ksql.function.udf.math.Round.round(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,069] INFO Adding function Round for method public java.lang.Double io.confluent.ksql.function.udf.math.Round.round(java.lang.Double,java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,072] INFO Adding function sign for method public java.lang.Integer io.confluent.ksql.function.udf.math.Sign.sign(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,078] INFO Adding function sign for method public java.lang.Integer io.confluent.ksql.function.udf.math.Sign.sign(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,079] INFO Adding function sign for method public java.lang.Integer io.confluent.ksql.function.udf.math.Sign.sign(java.lang.Long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,081] INFO Adding function sqrt for method public java.lang.Double io.confluent.ksql.function.udf.math.Sqrt.sqrt(java.lang.Double) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,088] INFO Adding function sqrt for method public java.lang.Double io.confluent.ksql.function.udf.math.Sqrt.sqrt(java.lang.Long) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,088] INFO Adding function sqrt for method public java.lang.Double io.confluent.ksql.function.udf.math.Sqrt.sqrt(java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,108] INFO Adding function elt for method public java.lang.String io.confluent.ksql.function.udf.string.Elt.elt(int,java.lang.String[]) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,112] INFO Adding function field for method public int io.confluent.ksql.function.udf.string.Field.field(java.lang.String,java.lang.String[]) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,114] INFO Adding function initcap for method public java.lang.String io.confluent.ksql.function.udf.string.InitCap.initcap(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,127] INFO Adding function mask_keep_left for method public java.lang.String io.confluent.ksql.function.udf.string.MaskKeepLeftKudf.mask(java.lang.String,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,129] INFO Adding function mask_keep_left for method public java.lang.String io.confluent.ksql.function.udf.string.MaskKeepLeftKudf.mask(java.lang.String,int,java.lang.String,java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,131] INFO Adding function mask_keep_right for method public java.lang.String io.confluent.ksql.function.udf.string.MaskKeepRightKudf.mask(java.lang.String,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,148] INFO Adding function mask_keep_right for method public java.lang.String io.confluent.ksql.function.udf.string.MaskKeepRightKudf.mask(java.lang.String,int,java.lang.String,java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,151] INFO Adding function mask for method public java.lang.String io.confluent.ksql.function.udf.string.MaskKudf.mask(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,152] INFO Adding function mask for method public java.lang.String io.confluent.ksql.function.udf.string.MaskKudf.mask(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,165] INFO Adding function mask_left for method public java.lang.String io.confluent.ksql.function.udf.string.MaskLeftKudf.mask(java.lang.String,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,166] INFO Adding function mask_left for method public java.lang.String io.confluent.ksql.function.udf.string.MaskLeftKudf.mask(java.lang.String,int,java.lang.String,java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,170] INFO Adding function mask_right for method public java.lang.String io.confluent.ksql.function.udf.string.MaskRightKudf.mask(java.lang.String,int) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,198] INFO Adding function mask_right for method public java.lang.String io.confluent.ksql.function.udf.string.MaskRightKudf.mask(java.lang.String,int,java.lang.String,java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,199] INFO Adding function replace for method public java.lang.String io.confluent.ksql.function.udf.string.Replace.replace(java.lang.String,java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,201] INFO Adding function split for method public java.util.List io.confluent.ksql.function.udf.string.SplitKudf.split(java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,203] INFO Adding function substring for method public java.lang.String io.confluent.ksql.function.udf.string.Substring.substring(java.lang.String,java.lang.Integer,java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,218] INFO Adding function substring for method public java.lang.String io.confluent.ksql.function.udf.string.Substring.substring(java.lang.String,java.lang.Integer) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,219] INFO Adding function url_decode_param for method public java.lang.String io.confluent.ksql.function.udf.url.UrlDecodeParamKudf.decodeParam(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,220] INFO Adding function url_encode_param for method public java.lang.String io.confluent.ksql.function.udf.url.UrlEncodeParamKudf.encodeParam(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,222] INFO Adding function url_extract_fragment for method public java.lang.String io.confluent.ksql.function.udf.url.UrlExtractFragmentKudf.extractFragment(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,227] INFO Adding function url_extract_host for method public java.lang.String io.confluent.ksql.function.udf.url.UrlExtractHostKudf.extractHost(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,252] INFO Adding function url_extract_parameter for method public java.lang.String io.confluent.ksql.function.udf.url.UrlExtractParameterKudf.extractParam(java.lang.String,java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,259] INFO Adding function url_extract_path for method public java.lang.String io.confluent.ksql.function.udf.url.UrlExtractPathKudf.extractPath(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,277] INFO Adding function url_extract_port for method public java.lang.Integer io.confluent.ksql.function.udf.url.UrlExtractPortKudf.extractPort(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,279] INFO Adding function url_extract_protocol for method public java.lang.String io.confluent.ksql.function.udf.url.UrlExtractProtocolKudf.extractProtocol(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,289] INFO Adding function url_extract_query for method public java.lang.String io.confluent.ksql.function.udf.url.UrlExtractQueryKudf.extractQuery(java.lang.String) (io.confluent.ksql.function.UdfLoader:143)
[2019-12-23 11:00:00,302] INFO Adding UDAF name=collect_list from path=internal class=class io.confluent.ksql.function.udaf.array.CollectListUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,338] INFO Adding UDAF name=collect_list from path=internal class=class io.confluent.ksql.function.udaf.array.CollectListUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,339] INFO Adding UDAF name=collect_list from path=internal class=class io.confluent.ksql.function.udaf.array.CollectListUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,341] INFO Adding UDAF name=collect_list from path=internal class=class io.confluent.ksql.function.udaf.array.CollectListUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,343] INFO Adding UDAF name=collect_list from path=internal class=class io.confluent.ksql.function.udaf.array.CollectListUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,368] INFO Adding UDAF name=collect_set from path=internal class=class io.confluent.ksql.function.udaf.array.CollectSetUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,369] INFO Adding UDAF name=collect_set from path=internal class=class io.confluent.ksql.function.udaf.array.CollectSetUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,370] INFO Adding UDAF name=collect_set from path=internal class=class io.confluent.ksql.function.udaf.array.CollectSetUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,371] INFO Adding UDAF name=collect_set from path=internal class=class io.confluent.ksql.function.udaf.array.CollectSetUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,372] INFO Adding UDAF name=collect_set from path=internal class=class io.confluent.ksql.function.udaf.array.CollectSetUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,374] INFO Adding UDAF name=avg from path=internal class=class io.confluent.ksql.function.udaf.average.AverageUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,389] INFO Adding UDAF name=avg from path=internal class=class io.confluent.ksql.function.udaf.average.AverageUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,396] INFO Adding UDAF name=avg from path=internal class=class io.confluent.ksql.function.udaf.average.AverageUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,398] INFO Adding UDAF name=histogram from path=internal class=class io.confluent.ksql.function.udaf.map.HistogramUdaf
(io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,399] INFO Adding UDAF name=sum_list from path=internal class=class io.confluent.ksql.function.udaf.sum.ListSumUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,400] INFO Adding UDAF name=sum_list from path=internal class=class io.confluent.ksql.function.udaf.sum.ListSumUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,407] INFO Adding UDAF name=sum_list from path=internal class=class io.confluent.ksql.function.udaf.sum.ListSumUdaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,409] INFO Adding UDAF name=WindowEnd from path=internal class=class io.confluent.ksql.function.udaf.window.WindowEndKudaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,410] INFO Adding UDAF name=WindowStart from path=internal class=class io.confluent.ksql.function.udaf.window.WindowStartKudaf (io.confluent.ksql.function.UdafLoader:71)
[2019-12-23 11:00:00,429] INFO UDFs can't be loaded as as dir /usr/ext doesn't exist or is not a directory (io.confluent.ksql.function.UserFunctionLoader:86)
[2019-12-23 11:00:00,450] INFO KafkaJsonDeserializerConfig values:
        json.fail.unknown.properties = true
        json.key.type = class io.confluent.ksql.rest.entity.CommandId
        json.value.type = class java.lang.Object
 (io.confluent.kafka.serializers.KafkaJsonDeserializerConfig:347)
[2019-12-23 11:00:00,451] INFO KafkaJsonDeserializerConfig values:
        json.fail.unknown.properties = true
        json.key.type = class java.lang.Object
        json.value.type = class io.confluent.ksql.rest.server.computation.Command
 (io.confluent.kafka.serializers.KafkaJsonDeserializerConfig:347)
[2019-12-23 11:00:00,834] INFO KafkaJsonSerializerConfig values:
        json.indent.output = false
 (io.confluent.kafka.serializers.KafkaJsonSerializerConfig:347)
[2019-12-23 11:00:00,890] INFO KafkaJsonSerializerConfig values:
        json.indent.output = false
 (io.confluent.kafka.serializers.KafkaJsonSerializerConfig:347)
[2019-12-23 11:00:08,406] INFO KsqlRestConfig values:
        access.control.allow.headers =
        access.control.allow.methods =
        access.control.allow.origin =
        authentication.method = NONE
        authentication.realm =
        authentication.roles = [*]
        authentication.skip.paths = [/v1/metadata, /v1/metadata/id, /healthcheck]
        compression.enable = true
        debug = false
        idle.timeout.ms = 30000
        ksql.healthcheck.interval.ms = 5000
        ksql.server.command.response.timeout.ms = 5000
        ksql.server.exception.uncaught.handler.enable = false
        ksql.server.install.dir = /usr
        ksql.server.preconditions = []
        ksql.server.websockets.num.threads = 5
        listeners = [http://0.0.0.0:8088]
        metric.reporters = []
        metrics.jmx.prefix = rest-utils
        metrics.num.samples = 2
        metrics.sample.window.ms = 30000
        metrics.tag.map = []
        port = 8080
        query.stream.disconnect.check = 1000
        request.logger.name = io.confluent.rest-utils.requests
        resource.extension.classes = []
        response.mediatype.default = application/json
        response.mediatype.preferred = [application/json]
        rest.servlet.initializor.classes = []
        shutdown.graceful.ms = 1000
        ssl.cipher.suites = []
        ssl.client.auth = false
        ssl.client.authentication = NONE
        ssl.enabled.protocols = []
        ssl.endpoint.identification.algorithm = null
        ssl.key.password = [hidden]
        ssl.keymanager.algorithm =
        ssl.keystore.location =
        ssl.keystore.password = [hidden]
        ssl.keystore.type = JKS
        ssl.protocol = TLS
        ssl.provider =
        ssl.trustmanager.algorithm =
        ssl.truststore.location =
        ssl.truststore.password = [hidden]
        ssl.truststore.type = JKS
        websocket.path.prefix = /ws
        websocket.servlet.initializor.classes = []
 (io.confluent.ksql.rest.server.KsqlRestConfig:347)
[2019-12-23 11:00:08,421] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:63)
java.io.IOException: Is a directory
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
        at java.io.InputStream.read(InputStream.java:101)
        at java.util.Properties$LineReader.readLine(Properties.java:435)
        at java.util.Properties.load0(Properties.java:353)
        at java.util.Properties.load(Properties.java:341)
        at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:564)
        at io.confluent.ksql.rest.server.ConnectExecutable.of(ConnectExecutable.java:41)
        at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:109)
        at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)```
McHardex commented 4 years ago

I got this to work by adding KSQL_KSQL_CONNECT_URL to my ksqldb-server environment

KSQL_KSQL_CONNECT_URL: http://connect:8083
basharat58 commented 4 years ago

@McHardex I am following https://ksqldb.io/quickstart.html (Windows 10) and have modified my docker file to be similar to yours above. It is:

---
version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-enterprise-kafka:5.3.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

  schema-registry:
    image: confluentinc/cp-schema-registry:5.3.1
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      - zookeeper
      - broker
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181

  connect:
    image: confluentinc/cp-kafka-connect:latest
    hostname: connect
    container_name: connect
    depends_on:
      - zookeeper
      - broker
      - schema-registry
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: broker:29092
      CONNECT_REST_ADVERTISED_HOST_NAME: localhost
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: ksql-connect-cluster
      CONNECT_OFFSET_STORAGE_TOPIC: ksql-connect-configs
      CONNECT_CONFIG_STORAGE_TOPIC: ksql-connect-topics
      CONNECT_STATUS_STORAGE_TOPIC: ksql-connect-statuses
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      #CLASSPATH: C:/Users/User01/Documents/confluent/share/confluent-hub-components      
      #CONNECT_PLUGIN_PATH: C:/Users/User01/Documents/confluent/share/confluent-hub-components      
      KSQL_KSQL_CONNECT_WORKER_CONFIG: C:/Users/basharat.hussain/GoogleTime/Kafka/connect.properties
    volumes: 
      - C:/Users/basharat.hussain/GoogleTime/Kafka/connect.properties

  ksqldb-server:
    image: confluentinc/ksqldb-server:0.6.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - broker
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker:29092
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_CONNECT_WORKER_CONFIG: C:/Users/basharat.hussain/GoogleTime/Kafka/connect.properties
      KSQL_KSQL_CONNECT_URL: http://connect:8083
    volumes: 
        - C:/Users/basharat.hussain/GoogleTime/Kafka/connect.properties

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.6.0
    container_name: ksqldb-cli
    depends_on:
      - broker
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

When I try to run docker-compose up it gives me an error: image

I see you are referencing (althoug I copied your file, I don't have confluent-hub-components on my machine:

CLASSPATH: C:/Users/User01/Documents/confluent/share/confluent-hub-components #CONNECT_PLUGIN_PATH: C:/Users/User01/Documents/confluent/share/confluent-hub-components

where can I download this from and how? Also do you know why I am getting this issue?

McHardex commented 4 years ago

@basharat58 My configuration has since changed. But however the error you're getting can be fixed. Your classpath should reference should look something like this:

CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components/,/data/connect-jars'

You do not have to modify that as it references the connect folder structure. Whatever jar you mount to connect will be installed in one of those directories so connect will check the three directories for any jar and use it.

To check your container folder structure: docker-compose exec -it container-name bash

in your case, that would be: docker-compose exec -it connect bash

you should use volumes to mount your jar. You can download them from confluent-hub. Alternatively, you can also download while starting up if you do not want to have them on your local machine. But that means you will have to download them every time you start up the containers. I hope my new connect config can guide you:

connect:
    image: confluentinc/cp-kafka-connect:5.3.1
    hostname: connect
    container_name: connect
    depends_on:
      - broker
      - schema-registry
    ports:
      - 8083:8083
    environment:
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_CUB_KAFKA_TIMEOUT: 300
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_ADVERTISED_HOST_NAME: 'connect'
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: connect-group-configs
      CONNECT_OFFSET_STORAGE_TOPIC: connect-group-offsets
      CONNECT_STATUS_STORAGE_TOPIC: connect-group-status
      CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      CONNECT_INTERNAL_KEY_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'
      CONNECT_INTERNAL_VALUE_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'
      CONNECT_LOG4J_ROOT_LOGLEVEL: 'INFO'
      CONNECT_LOG4J_LOGGERS: 'org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR'
      CONNECT_LOG4J_LOGGERS: 'org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR,org.eclipse.jetty.server=DEBUG'
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: '1'
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: '1'
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: '1'
      CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components/,/data/connect-jars'
      # External secrets config
      CONNECT_CONFIG_PROVIDERS: 'file'
      CONNECT_CONFIG_PROVIDERS_FILE_CLASS: 'org.apache.kafka.common.config.provider.FileConfigProvider'
    # volumes:
      # - C:\Users\User01\Downloads\connectors:/data/connect-jars
    #   - ./csv:/data/csv
    command: 
      # In the command section, $ are replaced with $$ to avoid the error 'Invalid interpolation format for "command" option'
      - bash 
      - -c 
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt debezium/debezium-connector-mysql:latest
        confluent-hub install --no-prompt confluentinc/kafka-connect-elasticsearch:7.5.1
        # confluent-hub install --no-prompt jcustenborder/kafka-connect-twitter:0.3.33
        confluent-hub install --no-prompt confluentinc/kafka-connect-jdbc:5.3.2
        # confluent-hub install --no-prompt confluentinc/kafka-connect-syslog:latest
        # confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.2
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run & 
        #
        sleep infinity

remember to comment out either the volume or the online jar download scripts depending on which one you want to go for.

basharat58 commented 4 years ago

You mentioned downloading the jar from confluent-hub, how do I go about downloading the Elasticsearch jar from confluent-hub in Windows 10?

McHardex commented 4 years ago

Just visit confluence hub. They have all the jars you need there

On Friday, February 21, 2020, basharat58 notifications@github.com wrote:

You mentioned downloading the jar from confluent-hub, how do I go about downloading the Elasticsearch jar from confluent-hub in Windows 10?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/confluentinc/ksql/issues/4127?email_source=notifications&email_token=AH24M27MW3HHAIL5RM3PHMTRD73BNA5CNFSM4J2BC2F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTFVVI#issuecomment-589716181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH24M22ES6GVWMDVORFWWYDRD73BNANCNFSM4J2BC2FQ .

-- Adebisi Oluwabukunmi J. Software Developer Mobile: +2348160601644 Email: bukunmiadebisi@gmail.com

MichaelDrogalis commented 3 years ago

We recently wrote a new, improved guide on how to use connector management.