search

ESGF / esg-search

ESGF Search Component
http://esgf.org/esg-search/
Other
8 stars 14 forks source link

Limit the maximum number of files returned in searches #119

Closed LucaCinquini closed 4 years ago

LucaCinquini commented 7 years ago

Who: Katharina

Hi Luca, all,

I found an issue in esg-search: When I search for all files (distributed search) and add a large offset (e.g. /esg-search/search?project=CMIP5&type=File&offset=99999999) esg-search refuses to work and breaks with "HTTP Status 500 - Connection refused". Solr Slave (port 8983) breaks at the same time as well.

This only happens for distributed search and core Files but not for the other cores (Datasets, Aggregation). I tried on two nodes, my production node which is running ESGF 2.4 and my ESGF 2.5. test node -- the same is happening for both. I have not tried on other production index nodes as I did not want to break anyone else's index.

I guess the issue is due to the large number of files but I'm not sure how to fix it? Would it be possible to prevent users from adding a large offset to the search constraints on file level search?

Thanks, Katharina

LucaCinquini commented 7 years ago

There's consensus about limiting to 10K files...

sashakames commented 4 years ago

Looks like this was implemented 2 years ago?