tomasfabian / ksqlDB.RestApi.Client-DotNet

ksqlDb.RestApi.Client is a C# LINQ-enabled client API for issuing and consuming ksqlDB push and pull queries and executing statements.
MIT License
97 stars 26 forks source link

Pull Query Server Pagination and Count of all Messages in a KTable #86

Closed fsimonovskiii closed 2 months ago

fsimonovskiii commented 2 months ago

Hi,

Had a few questions regarding achieving server pagination and counting messages in a KTable. They are closely related to each other which is why instead of opening couple issues I have put them in a single one for brevity.

  1. Is it possible to achieve full server pagination when doing a pull query on a materialized view? I have tried something like the following

    context.CreatePullQuery<JsonElement>(viewTableName)
    .Take(pageSize)
    .Skip(pageNumber) // this doesn't seem to be allowed
    .GetManyAsync()
    .ToListAsync();

    which doesn't let me define the Skip parameter on the server. If I place the Take and Skip after the GetManyAsync it just does the pagination after it polls all the data. I haven't found a Offset feature in the .NET client, so I was wondering if there is some other way?

  2. In addition to the first question, from the documentation I've read it states that there is no OrderBy supported because of possibility of records being on different partitions, but it was an older thread, so I was wondering are there any changes in this? What if we are dealing with a single partition topic? Probably one solution would be when creating the source table to do the ordering, but what if we need dynamic ordering?

  3. In order to return pagination metadata, we will need to have a total count of the messages in the KTable. One way I managed to get it to work is the following:

    
    await context.CreatePullQuery<dynamic>(viewTableName)
    .GetManyAsync()
    .CountAsync();

// make the second call to retrieve the data


I was wondering if there is a more efficient way to get the (filtered) data and the total `Count` in one call?

4. Last thing, is there a way to **see what is the ksql query that is being executed behind the scenes when we are using the .NET `KSqlDbContext`**? Do we always need to make sure that we create an `IPullable` and then do a `.ToQueryString()`?

Thanks in advance.
tomasfabian commented 2 months ago

Hello @fsimonovskiii, 1-3 Unfortunately, KSQL pull queries currently do not support any of the first three points.

4 You can configure logging to the console or a file to view the KSQL queries being used.