search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
209 stars 84 forks source link

Assistance with pagination limit #636

Open hyryuki opened 3 weeks ago

hyryuki commented 3 weeks ago

Hello, I am currently working with the API (GET/{branch}/concepts endpoint) for data analysis, and I have encountered a limitation regarding pagination. Specially, I am unable to retrieve all the data due to the pagination limits enforced on the endpoint. When using the unlimited pagination (searchAfter) I get the following error: "Unlimited pagination of the full concept representation is disabled in the deployment" Given that I need to analyze all available data, this limitation poses a significant challenge.

Questions: 1) is there any possibility to enable unlimited pagination for my deployment or is this a global restriction for all users? 2) If unlimited pagination cannot be enabled, do you have any alternative suggestions or best practices for retrieving all the data efficiently? For example, are there specific parameters or methods I could use to maximize the amount of data retrieved per request?

Thank you for your help, and I look forward to your response!

kaicode commented 3 weeks ago

Unlimited concept pagination can be enabled for your deployment using this configuration option:

--snowstorm.rest-api.allowUnlimitedConceptPagination=true

It's set to false in the default configuration: https://github.com/IHTSDO/snowstorm/blob/10.4.2/src/main/resources/application.properties#L24