search

Swirrl / ook

Structural search engine
https://search-prototype.gss-data.org.uk/
Eclipse Public License 1.0
6 stars 0 forks source link

Incremental back-off #65

Closed Robsteranium closed 3 years ago

Robsteranium commented 3 years ago

Even with really small page sizes we're seeing query timeouts as trawling through observations eventually pushes Stardog into a 15s GC pause.

If we see a org.eclipse.rdf4j.rio.RDFParseException with "Unexpected end of file" then it's probably a drafter time-out. Diagnosis with datadog images on slack.

We could attempt to back-off a bit - e.g. sleeping 30s then trying again. Possibly incrementally (i.e. multiple times with increasing wait time between) up until a limit.

Robsteranium commented 3 years ago

Also seeing Execution error (NullPointerException) at com.github.jsonldjava.core.JsonLdApi/filterNodes (JsonLdApi.java:1699) - this is presumably also the result of a timeout (if the EOF was actually at the beginning of the file, or at least after any number of complete statements that didn't include ?obs a qb:Observation criteria of the frame).