Failed to get json from webservice

[aainsa]

We have an error on our French ALA portal (OpenObs) that occurs frequently in production. The detail pages and the search result display this error after a one-minute waiting time:

• Failed to get json from web service (https://openobs.mnhn.fr/biocache-service/occurrence/be172f9e-39cd-406a-aee4-35b9e2ea2202). class java.io.IOException Got a non-200 response from server (404) for URL: https://openobs.mnhn.fr/biocache-service/occurrence/be172f9e-39cd-406a-aee4-35b9e2ea2202
• Failed to get json from web service (https://openobs.mnhn.fr/biocache-service/occurrences/search?q=lsid%3A249523&start=0&pageSize=20&sort=first_loaded_date&dir=desc&qc=&facets=datasetName%2CdynamicProperties_ownerInstitutionCode%2CrecordedBy%2CscientificName%2CtaxonConceptID%2Coccurrence_decade_i%2Craw_dataGeneralizations%2CdynamicProperties_raw_stateProvince&flimit=10). class java.net.SocketTimeoutException Read timed out

When looking at the logs of the BiocacheService, we find this error:

- Servlet.service() for servlet [mainDispatcher] in context with path [/biocache-service] threw exception [java.lang.IllegalStateException: getOutputStream() has already been called for this response] with root cause

When this occurs, we restart the biocacheservice Docker image to eliminate the error. We managed to reproduce the issue by making multiple requests on the site within a short period. We are using the version 3.0.24 of the BiocacheService. Have you ever encountered this error, or do you know where it might be coming from ?

[adam-collins]

Getting class java.net.SocketTimeoutException Read timed out after 60s for a call to https://openobs.mnhn.fr/biocache-service/occurrences/search? suggests a SOLR performance issue.

getOutputStream() has already been called for this response for a call to https://openobs.mnhn.fr/biocache-service/occurrences/search? suggests to me there is a bug with au.org.ala.biocache.web.CustomExceptionResolver. This bug may prevent logging of the actual cause.

With the information provided I would begin by looking for ZooKeeper connection drops and SOLR server load and memory. Logging SOLR GC has helped us identifying issues in the past.

[adam-collins]

I am assuming this is not longer an issue.