search

OneCricketeer / apache-kafka-connect-docker

[KAFKA-9774] Un-official Docker Image for Apache Kafka Connect
https://issues.apache.org/jira/browse/KAFKA-9774
Apache License 2.0
34 stars 15 forks source link

Starting a connector without using API #42

Closed ryanjclark closed 6 months ago

ryanjclark commented 6 months ago

It looks like this image expects connector configs to come via API or environment variables. It would be useful to supply worker.properties file instead.

Is that currently possible? Are you open to that feature?

OneCricketeer commented 6 months ago

Hi. Environment variables configure the connect-distributed.properties file (the worker properties), not any connector.

The ConnectDistributed process must start before any connector can be submitted to the API, as compared to running connect-standalone.sh script.

If this is a feature that is needed, I'd recommend using an init container pattern that submits an HTTP request, or look into Strimzi's KafkaConnect CRD (assuming you're using k8s)

ryanjclark commented 6 months ago

Got it. Thanks for clarification.

If I'm trying to avoid an HTTP request because I want to bundle 1 Cluster & 1 Connector into only configuration files, is there anyway to:

  1. Supply connect cluster configs to get it started (connect-distributed.properties)
  2. Then, automatically have 1 connector start given some-config-file.
OneCricketeer commented 6 months ago

This container currently calls the same function as connect-distributed.sh, which only accepts the worker properties, not a connector. POST-ing a config after the server starting would require an external process supervisor to ensure the server process remains running, but I am not sure that is possible with containers built with Jib

ryanjclark commented 6 months ago

OK thank you. I'll consider an init container! Cool project