redpanda-data / console

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
https://redpanda.com
3.8k stars 348 forks source link

Console strips unknown properties on /validate connectors call #1485

Open tomasz-sadura opened 3 days ago

tomasz-sadura commented 3 days ago

Use the following config for a connector (PostgreSQL Debezium):

{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "topic.prefix": "p1",
    "database.hostname": "xxx.eu-west-1.rds.amazonaws.com",
    "database.user": "postgres",
    "database.password": "xxx",
    "database.dbname": "postgres",
    "database.sslmode": "disable",
    "plugin.name": "pgoutput",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "header.converter": "org.apache.kafka.connect.storage.SimpleHeaderConverter",
    "name": "debezium-postgresql-connector-1",
    "converters": "someConverter",
    "someConverter.type": "com.some.converter.debezium.SomeConverter",
    "someConverter.tables": "public.test_table",
    "someConverter.tables.public.test_table.columns": "tags,schedule"
}

The properties someConverter.* are dynamic based on the converters value. These properties are not send to Kafka Connect with the /validate call, causing the connector to be unable to valiidate the config, as the required converter properties (someConverter.type) are not given. The UI hangs and user cannot create the connector.

Expected behaviour: All of the defined properties are sent with the /valdiate call.

tomasz-sadura commented 3 days ago
Screenshot 2024-10-25 at 10 15 32 Screenshot 2024-10-25 at 10 15 57