search

International-Data-Spaces-Association / InformationModel

The Information Model of the International Data Spaces implements the IDS reference architecture as an extensible, machine readable and technology independent data model.
Apache License 2.0
61 stars 35 forks source link

Pagingation in IDS #531

Open kragall opened 2 years ago

kragall commented 2 years ago

When Connectors query data from the Clearing House, they can either query for a single stored document or for all documents that are stored under a process_id. In the latter case this can result result in problems with too large responses, e.g. long response times or very large response messages.

In order to avoid these issues the Clearing House offers the Connectors the possibility to add query parameters to the Query-Endpoint, e.g. "?page=1&size=100", which will result in the Clearing House sending responses with at most 100 stored documents. With this solution arises the issue that the Connector needs to know, either what page is the last page that exists, or how many documents are stored in total.

A possible solution of this issue would be the introduction of a PagingResultMessage, that includes the needed attributes for the Connector as shown in this figure: PagingResultMessage

tmberthold commented 2 years ago

In the meantime, a presumably temporary workaround has been implemented in the EDC for Catalog-DescriptionRequests/Responses with large catalogs (Catalog with 5000 ContractOffers):

https://github.com/paullatzelsperger/DataSpaceConnector/blob/4320bbe922c422e2680dba3cf59c84a1fb239f5a/docs/developer/decision-records/2022-07-05/README.md

They use non-deterministic looping and send the CatalogRequest until the response no longer contains an offer.