search

geneontology / pipeline

Declarative pipeline for the Gene Ontology.
https://build.geneontology.org/job/geneontology/job/pipeline/
BSD 3-Clause "New" or "Revised" License
5 stars 5 forks source link

Add monitor to builds to prevent "failed" loads from going live #315

Closed kltm closed 1 year ago

kltm commented 1 year ago

This is a generalization of #309 , as we're now seeing it in the "main" pipelines:

18:21:48  2023-02-13 02:21:46,256 INFO  (GafSolrDocumentLoader:119) Processed 1000 bioentities at 5000 of 31811 and committing...
18:22:03  Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: org.apache.commons.httpclient.NoHttpResponseException: The server localhost failed to respond
18:22:03    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:475)
[...]
18:22:03    at owltools.cli.CommandRunnerBase.run(CommandRunnerBase.java:68)
18:22:03    at owltools.cli.CommandLineInterface.main(CommandLineInterface.java:12)
18:22:03  Caused by: org.apache.commons.httpclient.NoHttpResponseException: The server localhost failed to respond
18:22:03    at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1976)

Essentially, we want gross checks on Solr to detect a load failure.

kltm commented 1 year ago

While this is a good addition--I'm adding checks for both overall count and bioentity count--I think it is not the best way to go about this. Better would be 1) loader failing (long story why owltools does not fail correctly) or automated stats check before pinging human qc. Ideally both. That said, for the moment, this is an improvement.

kltm commented 1 year ago

Needs to be tested and propagated.

kltm commented 1 year ago

master pass; propagating.