SamCooper / COM_SPEC_RIDS

1 stars 0 forks source link

Archive query - impossible to limit the amount of retrieved data #90

Open dmarszk opened 7 years ago

dmarszk commented 7 years ago

It is not possible to limit the amount of retrieved data. Typical example would be retrieving all the COM Events that occurred in a whole year for browsing by the user. It could easily return mega or even gigabytes of data for an average mission.

I propose adding LimitFilter composite type to the COM::Archive service, extending the QueryFilter. LimitFilter would contain 2 fields:

long limit long offset

enabling pagination of the results. It would correspond to SQL query LIMIT and OFFSET functionality, described there: https://www.postgresql.org/docs/8.2/static/queries-limit.html

PostgreSQL manual states that the use of LIMIT and OFFSET is only reliable when the query uses ordering. Therefore, I suggest adding in the book that LimitFilter is not expected to be reliable when no sorting is applied. As typical implementation of COM Archive has SQL database on the backend.