500 errors when paging

[rsmith013]

Ok Richard.

I have come across this issue, which is only apparent for 'Ocean Colour' features with > 10,000 files Here is what I do in Python3.6:

retries = Retry( total=3, backoff_factor=4, status_forcelist=[500], method_whitelist=["GET"] )

opensearch_adapter = HTTPAdapter(max_retries=retries) with requests.Session() as session: session.mount('http://opensearch-test.ceda.ac.uk', opensearch_adapter) session.get(url) # multiple times, startPage incrementing until I get all files

Here is the output: Important part in red : Any ideas?

Ocean Colour : 6f891faf986349f792043ebac64f7938 Feature: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=1 totalResults: 10565 Feature URL: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=1 Feature URL: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=2 Feature URL: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=3 Feature URL: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=4 Feature URL: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=5 Feature URL: http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=6 HTTPConnectionPool(host='opensearch-test.ceda.ac.uk', port=80): Max retries exceeded with url: /opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=6 (Caused by ResponseError('too many 500 error responses',)) Response to 'http://opensearch-test.ceda.ac.uk/opensearch/request?parentIdentifier=6f891faf986349f792043ebac64f7938&httpAccept=application/geo%2Bjson&maximumRecords=2000&startPage=6' undefined Bytes: 831.3 GB

I get an intermittent 500 when trying this flow and it reports KeyError: No key 'info' https://github.com/cedadev/archive-opensearch/blob/f8b8813961d3b0d1017420f8a62511c7b945577a/django_opensearch/opensearch/backends/elasticsearch/facets/cci.py#L90

If I try to catch a KeyError, then the call passes. The calls which 500 take longer than the successful calls, so I wonder if it is a timeout in the elasticsearch layer which means the _source does not contain an info key. Look into handling timeouts in the ES layer.