Cannot get total number of query results

[TakashiAbe0309]

Hi, All. I am trying to get historical data from Quantum Leap API in python.

Is your feature request related to a problem? Please describe.

I could pagination by using offset and limit queries. However, I don't know how many results will be got in total for the query results and I have to shift the offset by the limit and get it until there is no data. This method is very inconvenient if you want to know the total number of matched results.

Describe the solution you'd like

Orion provides a very useful header for this. https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/user/pagination.md#pagination Like Orion, by including options=count in the API query, Fiware-Total-Count is returned in the response header.

Thank you.

[c0c0n3]

Hi @TakashiAbe0309!

Thanks for reporting this.

This method is very inconvenient if you want to know the total number of matched results.

Yep, it is :-)

I'm jotting down some notes about this total count feature so we can think about what to do when we tackle it.

The problem with pagination and total count

We put some thought into improving queries in general

• https://github.com/smartsdk/ngsi-timeseries-api/wiki/The-Mother-of-all-Queries

In particular your issue

I don't know how many results will be got in total for the query results and I have to shift the offset by the limit and get it until there is no data

is part of a broader problem involving hidden data loss/partial results and the limit not being honoured

• https://github.com/smartsdk/ngsi-timeseries-api/wiki/Issues

This is a hard nut to crack right now for us because QuantumLeap doesn't snapshot query pages, so there's no guarantee the results fetched by a page will be consistent w/r/t the initial query---see this comment about it. For example, say you split a query q for an attribute a in 2-value-wide pages and that, at the time you run it, q selects the values below:

a = [1, 2, 3, 4]

Now the total count here is 4 and there are two pages

p1 = [1, 2]
p2 = [3, 4]

But after fetching p1, the database state changes and now q matches more a-values:

a = [0, 1, 2, 3, 4]

At this point fetching the second page brings back [2, 3] instead of [3, 4]. Notice how the value 2 gets returned twice! The implication is that any calculations the client does on a paginated result set could be out of whack. For example, taking the sum of the values would yield 1 + 2 + 2 + 3 = 8.

Practical workaround

While we think about how to implement a total count, there's a workaround clients could use right now. If you can assume values get only appended to the DB (normally this should be the case since we're dealing with historical records here), then adding a (not-in-the-future) toDate parameter to the query should make pagination consistent---see #417 about it. In this scenario you could ask for the total result count upfront using the aggrMethod param as in

...? toDate=2021-01-01T00:00:00+00:00 & aggrMethod=count & ... <rest of the query>

and then paginate the query

...? toDate=2021-01-01T00:00:00+00:00 & offset=0 & limit=100 & ... <rest of the query>

[chicco785]

@TakashiAbe0309 thanks for you feedback! While I understand your point, I am not sure it is an ideal solution from two points of view:

1. Performance (one more query to issue)
2. Data completeness illusion (in a timeseries, if you don't limit the time span, it is likely that new data come in, and the counter issued in the first response would not be correct anymore) (this is exactly the case @c0c0n3 explained)

[TakashiAbe0309]

@c0c0n3 @chicco785 Thank you for your quick reply. I was able to get the total number of query results by adding aggrMethod=count to the query parameters.