(pull)-query a joined table

[ShahOdin]

I have asked the question on stack-overflow: https://stackoverflow.com/questions/60123191/ksql-pull-query-a-joined-table

• Am I making a mistake?
• Is the documentation unclear, or is it just me?

[ShahOdin]

@vpapavas @agavra could you please let me know if there is a workaround in the mean-time? Are pull queries not supported in the 0.5.x versions? I can see references to it all over the place and I am very confused as to why I can't seem to do it!

is it a problem with my streams/tables? is it a problem with the control center editor?

[mjsax]

Currently, ie, ksqlDB 0.6.0, only stream aggregation queries that return a table allow to query the result table.

For a table-table join, the result is not materialized into a local store, but only a changelog stream is produced and written to the result topic that corresponds to the result table.

[ShahOdin]

@mjsax Thanks.

• I am testing with the latest version available on docker, namely 0.5.x

• I have the same problem on both tables and streams. All the combinations below.

  • table created out of kafka topic
  • stream created out of kafka topic
  • table created out of table-table join
  • stream created out of stream-stream join

  fail to return pull queries.

[mjsax]

Thanks. I just double checked the KLIP. Seems, the backward compatibility was only done for persistent queries, but not for transient queries: https://github.com/confluentinc/ksql/blob/master/design-proposals/klip-8-queryable-state-stores.md#backward-compatibility

[ShahOdin]

@mjsax Thanks. so what are the actions to take from this? and in the mean time, Is there a workaround for this? so I can present a demo to my team?

[ShahOdin]

@mjsax to clarify,

• I was doing my tests with the cp-all-in-one docker-compose file , ie using: confluentinc/cp-ksql-server:5.4.0.

• according to this the point-in-time queries are now supported from 0.6.0. The documentation too, suggest this is already available. (ie, here) However, your comment above suggests otherwise. Also this document implies that this is to be available in the 0.8.0 relase.

• By looking at above, I also noticed there exists an image called confluentinc/ksqldb-server which is now at version 0.7.0 which presumably supports the said feature.

• I think you misdiagnosed my problem but I am glad you spotted a missed feature.

• how do the two images confluentinc/cp-ksql-server and confluentinc/ksqldb-server differ, other than the supposed env-var setup for integration with the other cp-x components. shouldn't they be in sync?

• Is there an unofficial equivalent of cp-all-in-one's docker-compose file that uses confluentinc/ksqldb-server instead?

• The docker-compose file advertised on ksqldb.io doesn't have schema-registry and connect integration which I would like to use at my demo. Can you help me put together a setup, consisting of at least: [broker, zookeeper, schema-registry, connect] but ideally a full-on cp-all-in-one setup where everything works together?

[mjsax]

Well, pull queries still have limitations. It's maybe too subtle in the docs but the example says:

SELECT select_expr [, ...]
  FROM aggregate_table

(1) pull queries over STREAMS (what you mention above as stream created out of kafka topic and stream created out of stream-stream join) do not really make sense to begin with, because a stream is stateless -- there is nothing to be queried.

(2) Atm, pull quries are only supported on TABLES that are the result of an aggregation query (cf. above FROM aggregate_table -- I agree, it's quite subtle). The two types of tables you mentioned above (table created out of kafka topic and table created out of table-table join) are not yet queryable, because neither of both tables is actually materialized in a local state store, but only the result changelog topic is written into the corresponding output topic. It's a known (maybe not well documented) limitation that is still in 0.8.0 release (and it's unclear in which release it will be fixed atm).

[big-andy-coates]

This will be address by https://github.com/confluentinc/ksql/issues/5652. Closing this ticket as a duplicate.