Allow setting auto.offset.reset independent of a specific query

[kaiwaehner]

I would like to set

SET 'auto.offset.reset'='earliest'; SET 'auto.offset.reset'='latest';

I hoped it would work like the following with the generic 'client.ksql' command: client.ksql('SET \'auto.offset.reset\'=\'earliest\'');

It works well from KSQL CLI, so I think it is not a KSQL issue, but a limitation in the Python wrapper?

I found this related ticket: https://github.com/bryanyang0528/ksql-python/issues/39 Though, I think it is much more cumbersome if you have to add this property to every query instead of doing it just once before executing several queries afterwards.

[bryanyang0528]

I found this page https://docs.confluent.io/current/ksql/docs/developer-guide/api.html#post--ksql

POST /ksql HTTP/1.1
Accept: application/vnd.ksql.v1+json
Content-Type: application/vnd.ksql.v1+json

{
  "ksql": "CREATE STREAM pageviews_home AS SELECT * FROM pageviews_original WHERE pageid='home'; CREATE STREAM pageviews_alice AS SELECT * FROM pageviews_original WHERE userid='alice'",
  "streamsProperties": {
    "ksql.streams.auto.offset.reset": "earliest"
  }
}

The API call of setting properties is different from SQL statement. I think I should create a new API function for it.

[zaidyahya]

Hi @bryanyang0528 , were you able to add this?

If not, just to confirm that without it, a query through the client would only show the new records that are added to the table/stream, correct? There's no way to run a select query for e.g. that would show records from the beginning?

[bryanyang0528]

@zaidyahya Hi I just test the KSQL api to the latest version of KSQL. I will work on this task again.

[harlev]

See this note On this page https://docs.ksqldb.io/en/0.10.1-ksqldb/developer-guide/ksqldb-rest-api/streaming-endpoint/

[bryanyang0528]

@harlev thanks for the reference