Develop plan for handling resource-hogging queries

RENCI-AUTOMAT / Automat-server

MIT License

0 stars 0 forks source link

Develop plan for handling resource-hogging queries #10

Closed cbizon closed 2 years ago

cbizon commented 3 years ago

This query times out: { "message": { "query_graph": { "nodes": { "a": { "category": "biolink:ChemicalSubstance", "id": "CHEBI:31823" }, "b": { "category": "biolink:Gene" } }, "edges": { "ab": { "subject": "a", "object": "b", "predicate": "biolink:decreases_activity_of" } } }, "knowledge_graph": { "nodes": {}, "edges": {} }, "results": [] } }

I tried a cypher query to just return counts, but it timed out too.

YaphetKG commented 3 years ago

Issue was related to neo4j out of memory . Added 2G more to the instance. Currently CTD neo4j is at 4G. Probable cause might have been unbound query. For preventing this from happening in the future I thinking we can:

Add validation for Plater to run query iff its bound (?)
Add liveliness probe for neo4j in our K8s deployments by detecting GC world pause errors and OOM errors and restart the server automatically (?)

cbizon commented 3 years ago

Hmm, that makes sense. I think I accidentally sent an unbound query first due to a bug in the query generator. I think that there's also a way to put some kind of timeout on a query?

I do think that the validation makes sense.

Modified issue title to reflect next steps

YaphetKG commented 2 years ago

We have query timeout for neo4j set to 1600s by default for all deployments.