Feature Request: Some improvements to `kafka connect`

[cobolbaby]

• The number of list elements displayed exceeds the paging setting when the webpage first loads
• Table overflow
• The filter option box does not support state storage
• Not support sorting and filtering by connector status
• Not support jumping from the connector to its tasks

[rikimaru0345]

1, 3, 4: will fix 👍 2: I'll add a maximum width for the column (full text will be shown with a popup on hover then) 5: By "from connector" you probably mean the connector details thing (with the connector config editor and the tasks list below it), right? However there is no "task details" view currently. I can add one if that would be useful to you. But what exactly would you like it to show other than the stuff in the table below the config editor?

Thanks for the feedback! 😄

[cobolbaby]

But what exactly would you like it to show other than the stuff in the table below the config editor?

Sometimes we will notice that even though the connector is running, its task has failed. At this time, we need a restart button to restart a task individually. Just like vue-connect:

[weeco]

@cobolbaby as per usual awesome feedback. We'll make sure to address this before we will release the new version. Thanks again!

[weeco]

Hey @cobolbaby , I think we addressed all your points (except table overflow) - if we missed something please let us know. We also added some logos for each connector. Would be nice if you could provide another screenshot once you are running the latest version - you seem to have a decent environment with lots of connectors :). You can find the tasks inside of the connectors details page and now we added the possibilty to restart the task.

Additionally we added a list of installed plugins inside of each Kafka connect cluster. You can find that in the newly added cluster details page.

Looking forward to your feedback :). Latest docker tag is master-edebcedf

[cobolbaby]

if you could provide another screenshot once you are running the latest version

The front-end basically meets the needs, but some details need to be confirmed. I don’t know if it meets expectations.

• The filter condition can only show up when the user click the filter box. Does the list component support to display the filter conditions by default?
• The pagination list does not change with the input filter conditions.
• If I want to add the logo of other connectors, how do I configure it?

[rikimaru0345]

The front-end basically meets the needs, but some details need to be confirmed. I don’t know if it meets expectations.

Keep in mind I've only added those changes in a quick-and-dirty way for now. There'll definitely be a polishing phase to improve the design of those Kafka Connect pages and iron out the bugs regarding pagination, search etc.

The filter condition can only show up when the user click the filter box. Does the list component support to display the filter conditions by default?

Can you elaborate on that? What do you mean by filter conditions? Some custom syntax maybe, like class:"abc"?

Or do you simply mean that the search-box should be visible by default / when the table loads.

Currently the searchbox checks for the string/regex in all the text columns (name, class, type, state, cluster).

But I'm fully aware that there is still a lot that could/should be improved, for example:

• Making it obvious to the user what columns get searched
• Highlighting matched substrings in the table
• Maybe some custom syntax like described above
• An easier way to filter the other columns if they are "enum like" values
• etc..

The pagination list does not change with the input filter conditions.

Yep, that's a known bug. I'll get to it soon.

If I want to add the logo of other connectors, how do I configure it?

Currently there is no way to manually configure that. Speaking purely from a frontend standpoint, I could definitely add a dialog to add/customize the names and images for the connectors. But that would then be saved in the browser only, which I very much assume isn't really all that helpful. We could probably make it configurable in the backend config-file; that'd not take much time I think (but then there'd be no frontend UI to change this config, since that would imply the changes are persisted somewhere in the backend). @weeco what do you think about that?

[weeco]

If I want to add the logo of other connectors, how do I configure it?

I think it's good enough to rely on code contributions here. Sooner or later we should have a pretty complete list. Logos have to be added here: https://github.com/cloudhut/kowl/tree/master/frontend/src/assets/connectors and a mapping between connector class name (prefix) and the logo here: https://github.com/cloudhut/kowl/blob/master/frontend/src/components/pages/connect/helper.tsx (see connectormetadata)

[cobolbaby]

Or do you simply mean that the search-box should be visible by default / when the table loads.

@rikimaru0345 Yes, this is exactly what I want.

But I'm fully aware that there is still a lot that could/should be improved, for example:

Awesome.

If I want to add the logo of other connectors, how do I configure it?

No matter where the configuration is stored, scalability is the primary consideration, and it should be convenient for contributors to add new items. Currently when adding a new connector configuration, contributors need to modify the front-end code. I think the front-end should traverse and load the connector configuration from a configuration list which can be stored in a file on the front-end or back-end.

[weeco]

I think the front-end should traverse and load the connector configuration from a configuration list which can be stored in a file on the front-end or back-end.

Can you clarify that? What do you mean by loading the connector configuration. The only per connector configuration that is currently coded in the frontend is a mapping between a connector class name and the logo and I think it's fair to keep that as part of the code, rather than making it configurable dynamically on the user side.

If you mean creating new connectors via Kowl, there's a separate issue for that (#249 ) which you already saw though

[cobolbaby]

If you mean creating new connectors via Kowl, there's a separate issue for that (#249 ) which you already saw though

No.

What do you mean by loading the connector configuration

Migrate the configuration from .tsx to a separate .json file

[rikimaru0345]

@cobolbaby can you test the latest version? I'll add filtering for "enum columns" in all other tables tomorrow.

[cobolbaby]

I only saw the icon changes in the latest version. The feature improvements previously mentioned have not been implemented.

In addition, I find that the connector list does not support filtering now, I don't know if it is intentional.

[rikimaru0345]

Should be fixed with https://github.com/cloudhut/kowl/commit/72f45e335020cb711ee3d30cd49570247e681636 All columns in all kafka-connect pages (and most other tables) should have filters for their enum columns now.

Currently the tables save their search-text, but they don't save the filters you selected or the sort-order of the columns. I'll implement that soon.

After that I'll probably move on to other features, unless you have any other requests or suggestions; so let me know if there is anything 😄

Also, there was the question of which columns exactly the text/regex-search searches in: I haven't been able to figure out a good way to represent that. Currently every table is different, some tables look for matches only in the "main" (name) column, others also include other columns... :/

[cobolbaby]

@rikimaru0345 There is indeed a necessary feature, that is, how to view the error message when a connector fails.

[weeco]

@cobolbaby As far as I know that kind of information is not accessible via the exposed REST API from a Connect cluster or am I wrong? I think you are supposed to look at the log messages of the connect cluster in that case, until Apache implements something for that (I wish that was possible too)

[cobolbaby]

When the connector fails, the api /connectors?expand=info&expand=status will return the trace field, including the error stack of the connector.

    ...
    "rabbitmq-source-ams-reflow": {
        "info": {
            "name": "rabbitmq-source-ams-reflow",
            "config": {
                "connector.class": "com.github.themeetgroup.kafka.connect.rabbitmq.source.RabbitMQSourceConnector",
                "rabbitmq.password": "xxx",
                "tasks.max": "1",
                "rabbitmq.network.recovery.interval.ms": "10000",
                "rabbitmq.queue": "AMS-T-Reflow",
                "rabbitmq.username": "xx",
                "rabbitmq.exchange": "AMSMessageExchangeTPMS",
                "rabbitmq.virtual.host": "xx",
                "rabbitmq.prefetch.count": "500",
                "rabbitmq.port": "5672",
                "rabbitmq.routing.key": "*.*",
                "rabbitmq.queue.ttl": "600000",
                "name": "rabbitmq-source-ams-reflow",
                "rabbitmq.host": "xxxx",
                "kafka.topic": "AMS-Reflow-Rabbit2kafka",
                "message.converter": "com.github.themeetgroup.kafka.connect.rabbitmq.source.data.StringSourceMessageConverter",
                "rabbitmq.automatic.recovery.enabled": "true",
                "key.converter": "org.apache.kafka.connect.storage.StringConverter"
            },
            "tasks": [
                {
                    "connector": "rabbitmq-source-ams-reflow",
                    "task": 0
                }
            ],
            "type": "source"
        },
        "status": {
            "name": "rabbitmq-source-ams-reflow",
            "connector": {
                "state": "RUNNING",
                "worker_id": "kafka-connect-3:8083"
            },
            "tasks": [
                {
                    "id": 0,
                    "state": "FAILED",
                    "worker_id": "kafka-connect-3:8083",
                    "trace": "org.apache.kafka.connect.errors.ConnectException: java.io.IOException\n\tat com.github.themeetgroup.kafka.connect.rabbitmq.source.RabbitMQSourceTask.start(RabbitMQSourceTask.java:90)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.io.IOException\n\tat com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)\n\tat com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)\n\tat com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:147)\n\tat com.rabbitmq.client.impl.ChannelN.exchangeDeclare(ChannelN.java:783)\n\tat com.rabbitmq.client.impl.recovery.AutorecoveringChannel.exchangeDeclare(AutorecoveringChannel.java:251)\n\tat com.rabbitmq.client.impl.recovery.AutorecoveringChannel.exchangeDeclare(AutorecoveringChannel.java:241)\n\tat com.rabbitmq.client.impl.recovery.AutorecoveringChannel.exchangeDeclare(AutorecoveringChannel.java:221)\n\tat com.github.themeetgroup.kafka.connect.rabbitmq.source.RabbitMQSourceTask.start(RabbitMQSourceTask.java:75)\n\t... 8 more\nCaused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'AMSMessageExchangeTPMS' in vhost 'ams': received 'false' but current is 'true', class-id=40, method-id=10)\n\tat com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)\n\tat com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)\n\tat com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)\n\tat com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293)\n\tat com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141)\n\t... 13 more\nCaused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'AMSMessageExchangeTPMS' in vhost 'ams': received 'false' but current is 'true', class-id=40, method-id=10)\n\tat com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:517)\n\tat com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:341)\n\tat com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:182)\n\tat com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:114)\n\tat com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:739)\n\tat com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:47)\n\tat com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:666)\n\t... 1 more\n"
                }
            ],
            "type": "source"
        }
    },
    ...

[weeco]

@cobolbaby Found it, @rikimaru0345 only has to add it into the frontend and we got that as well. Thanks for the awesome hint again

[cobolbaby]

The status field of the task list does not match the actual status.

kowl version: master-d782eab

[rikimaru0345]

Yeah, that column was completely wrong actually. Text in the popup wasn't formatted/capitalized either. Should be fixed now! https://github.com/cloudhut/kowl/commit/10776b81e45abfa716548d171d7f7e1326bf8553

[cobolbaby]

LGTM. Thank you for your support.