SOLR is down

[kjgarza]

OAI is not working

Front conversations

[kjgarza]

changed instance

[kjgarza]

Expected Behaviour

Solr should be working during indexing and when users harvest doi metadata.

Current Behaviour

Solr crashes while users are harvesting doi metadata in oai

Steps to Reproduce

Make this query

https://solr.datacite.org/api?q=*:*&rows=50&start=9555400&sort=updated asc&fq=updated:[0001-01-01T00:00:00Z TO 9999-12-31T23:59:59Z]&qt=/api&wt=javabin&version=2

Context (Environment)

it occured twice in jan 9 at the same time of a SOLR indexing and another time during harvesting doi on jan 13 (there were no logs because oai moved to fargate ithink).

some of the queries seem from only one client TIB.AWI

/oai?verb=ListRecords&resumptionToken=[1547270829286-1547343979088],0001-01-01T00:00:00Z,9999-12-31T23:59:59Z,[50-5750],TIB.AWI,oai_datacite

and some times the y seem to go into very deep pages even when small

request: https://solr.datacite.org/api?q=*:*&rows=50&start=9555400&sort=updated asc&fq=updated:[0001-01-01T00:00:00Z TO 9999-12-31T23:59:59Z]&qt=/api&wt=javabin&version=2

Seems to be going to document 477,770,000

Hypothesis

deep paginating is killing solr.

currently TIB.AWI doesn't seem to realise the service is down and it keep deepaginating even if they get errors.

By deterring deep pagination the index wont die.

Detailed Description

Block deep paginating for solr.

Possible Implementation

reverse https://github.com/datacite/search/commit/bc0e86e7e7a031efc45991b47e139ecd73cc9427#diff-417dfcc57c4e2ca3fda1f763a90837fe

[kjgarza]

it solved the issue. We had to reindex both instances

[kjgarza]

For reference:

to release a new version of the Solr container one needs to change the tag version in the vpc folder https://github.com/datacite/mastino/blob/1acb68cc0daca8e2c9ed4ec0574f21cfe8315b0b/prod-eu-west/vpc/_search.auto.tfvars#L7

but to change the instance to be shown in the elb the change has to be made in search folder