Request for Clarification on Unlimited Pagination for {branch}/concepts Endpoint

[paloma-cito]

Hello,

I am developing an application using Snowstorm and have encountered an issue with the GET /{branch}/concepts endpoint. Specifically, I am running into the 10,000 results limit, which is problematic as I often need to process ECL queries that return more than 10,000 results.

In the documentation, I noticed that there is an option to disable this limitation for another endpoint (GET /browser/{branch}/concepts) using the following configuration:

snowstorm.rest-api.allowUnlimitedConceptPagination

Is a similar option available for the GET /{branch}/concepts endpoint? If so, could you provide guidance on how to enable it?

Any assistance you can provide would be greatly appreciated.

[kaicode]

The Snowstorm native API that you are using does not allow pagination beyond 10,000 results when using the offset parameter. However it does allow unlimited pagination via the searchAfter parameter. We have this parameter as a workaround for a restriction in Elasticsearch. Elasticsearch that is not able to paginate using offset without reading all the results from previous pages into memory and that gives poor performance.

To paginate using searchAfter:

• request the first page of results in the normal way, setting the limit parameter as required to specify the page size
• take the searchAfter string from the end of the first page and set it as a request parameter to load the subsequent page
• repeat as needed

For example the first three pages from the Jan 2024 International Release could be loaded like this:

• https://browser.ihtsdotools.org/snowstorm/snomed-ct/MAIN/2024-01-01/concepts?activeFilter=true&limit=5
• https://browser.ihtsdotools.org/snowstorm/snomed-ct/MAIN/2024-01-01/concepts?activeFilter=true&limit=5&searchAfter=WyI5OTk0ODA1NjEwMDAwODcxMDAiXQ==
• https://browser.ihtsdotools.org/snowstorm/snomed-ct/MAIN/2024-01-01/concepts?activeFilter=true&limit=5&searchAfter=WyI5OTkxMDA4Il0=

The allowUnlimitedConceptPagination parameter you mentioned only blocks the /browser/{branch}/concepts endpoint. This is disabled by default to prevent anyone extracting the complete set of concept definitions from the public browser.

[paloma-cito]

Hi,

Thank you very much for your prompt and thorough response. Your solution solved my issue, and I truly appreciate your help!

Best regards,

Paloma