confluentinc / kafka-tutorials

Tutorials and Recipes for Apache Kafka
https://developer.confluent.io/tutorials
Apache License 2.0
16 stars 90 forks source link

No output : (ksqlDB Tutorial) How to find distinct values in a stream of events #942

Open bluedog13 opened 3 years ago

bluedog13 commented 3 years ago

For the tutorial below https://kafka-tutorials.confluent.io/finding-distinct-events/ksql.html

A table is created as below

CREATE TABLE DETECTED_CLICKS AS
    SELECT
        IP_ADDRESS AS KEY1,
        URL AS KEY2,
        TIMESTAMP AS KEY3,
        AS_VALUE(IP_ADDRESS) AS IP_ADDRESS,
        AS_VALUE(URL) AS URL,
        AS_VALUE(TIMESTAMP) AS TIMESTAMP
    FROM CLICKS WINDOW TUMBLING (SIZE 2 MINUTES, RETENTION 1000 DAYS)
    GROUP BY IP_ADDRESS, URL, TIMESTAMP
    HAVING COUNT(IP_ADDRESS) = 1;

Then a stream is declaring with the explanation saying - "statement declares a stream on top of the DETECTED_CLICKS changelog topic, defining only the value columns we’re interested in"

CREATE STREAM RAW_DISTINCT_CLICKS (IP_ADDRESS STRING, URL STRING, TIMESTAMP STRING)
    WITH (KAFKA_TOPIC = 'DETECTED_CLICKS',
          PARTITIONS = 1,
          FORMAT = 'JSON');

But in reality, a new topic by name "DETECTED_CLICKS" is created that has no idea about "DETECTED_CLICKS" changelog, and as a result, this topic contains no messages.

For the same reason, the subsequent STREAM "DISTINCT_CLICKS" based on the above stream does not display any output as expected.

bluedog13 commented 3 years ago

The topic name the table creates usually has the "ksql.service.id" as the prefix to the topic name.

In my case

"ksql.service.id" = "pksqlc-zm127"

and hence the topic name created for the table is "pksqlc-zm127DETECTED_VEHICLE_LOCATION"

In the tutorial, there is no "ksql.service.id" configuration in the docker-compose file and perhaps due to the very same reason, the underlying topic that gets created is "DETECTED_VEHICLE_LOCATION" - the reason why I am not able to see output in my setup.

Can this explanation be confirmed?

Thanks.