[Bug]: Error when running the debezium postgres example

sebymiano commented 5 months ago

Steps to reproduce

Hello everyone, I am trying to run the debezium-postgres-example under the pathway-examples repo, however I get the error shown in the log below. I am just running the make command under the debezium-postgres-example; all the contains start correctly, but for some reason pathway is not able to parse the received message from Kafka.

Any idea of how this can be solved?

Relevant log output

Imports OK!
Starting Pathway:
<pathway.Table schema={'value': <class 'int'>}>
[2024-01-27T17:48:28]:INFO:Preparing Pathway computation
[2024-01-27T17:48:28]:ERROR:librdkafka: Global error: UnknownTopicOrPartition (Broker: Unknown topic or partition): Subscribed topic not available: dbserver1.public.values: Broker: Unknown topic or partition
[2024-01-27T17:48:28]:ERROR:There had been an error processing the row read result: Message consumption error: UnknownTopicOrPartition (Broker: Unknown topic or partition)
[2024-01-27T17:48:28]:INFO:KafkaReader-0: 0 entries (1 minibatch(es)) have been sent to the engine
[2024-01-27T17:48:28]:INFO:PsqlWriter-1: Done writing 0 entries, time 1706377708732. Current batch writes took: 0 ms. All writes so far took: 0 ms.
[2024-01-27T17:48:28]:INFO:FileWriter-0: Done writing 0 entries, time 1706377708732. Current batch writes took: 0 ms. All writes so far took: 0 ms.
[2024-01-27T17:48:41]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:41]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:42]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:42]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:43]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:43]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:44]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:45]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:45]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:46]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:46]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:47]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:47]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:48]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:48]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:49]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:49]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:50]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:50]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:51]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:51]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:52]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload
[2024-01-27T17:48:53]:ERROR:Read data parsed unsuccessfully. received message doesn't have payload

What did you expect to happen?

I would expect to see the same output as the one shown in the tutorial page: https://pathway.com/developers/user-guide/exploring-pathway/realtime-analytics-with-cdc

Version

0.7.10

Docker Versions (if used)

24.0.7

OS

Linux

On which CPU architecture did you run Pathway?

x86-64

olruas commented 5 months ago

Hello @sebymiano. Thanks a lot for discovering this problem. Our connector seems to fail to connect because there is no primary key in the SQL database. A fix is coming in the next release. Meanwhile, you can fix this by changing the SQL config file /sql/init-db.sql to add a primary key in the values table:

CREATE TABLE IF NOT EXISTS values (
    value integer NOT NULL,
    id SERIAL PRIMARY KEY
);

Can you tell me whether it fixes your issue?

sebymiano commented 5 months ago

Hello @olruas, thanks for the fix! It works now :)

olruas commented 5 months ago

Awesome :) Thanks again for reaching out for this. Don't hesitate to contact us on Discord if you have any questions!

pathwaycom / pathway