Running reasoner on models on noctua-dev causes nodes and edges to disappear from graph editor

[vanaukenk]

I'm testing the latest updates on noctua-dev and all looks okay except that when I ran the reasoner in the graph editor on a test model, all of the nodes and edges disappeared from view.

I tried zooming out just in case they went hiding somewhere, but I didn't see them, and turning the reasoner off doesn't bring them back.

The corresponding annotations in the table on the form still look okay.

Here's the model:

http://noctua-dev.berkeleybop.org/editor/graph/gomodel:5ea8c44700000037?model_id=gomodel:5ea8c44700000037

[ukemi]

Sometimes if you click on the OWL export, the graph will show up again. @goodb showed me this trick a while ago.

[vanaukenk]

Good to know, but things shouldn't disappear from the editor!!

[vanaukenk]

And if I try to save the model from the graph editor, I get an error message that says the model has no individuals:

[kltm]

@vanaukenk Something has gone wrong with Minerva and it is no longer returning individuals.

[kltm]

I would guess it's the same issue as what is causing https://github.com/geneontology/noctua/issues/671

[kltm]

@vanaukenk Where did the URL http://noctua-dev.berkeleybop.org/editor/graph/gomodel:5ea8c44700000037?model_id=gomodel:5ea8c44700000037 come from? That extra identifier is not needed and should probably be removed (for the time being).

[vanaukenk]

@kltm I copied the URL from my Chrome browser window that was displaying the graph editor.

[kltm]

@vanaukenk Assuming that you came from the form or search, we may need to add an issue to remove the unneeded extra argument--that will be confusing.

[kltm]

@goodb On dev, after a code refresh and restart, for this model, there seems to be some issue that only individuals are returned, no edges. Before the restart, the opposite was true--only edges, no individuals.

The current minerva command that I'm running on is: java -Xmx64G -cp ../minerva/minerva-cli/bin/minerva-cli.jar org.geneontology.minerva.server.StartUpTool --use-golr-url-logging --use-request-logging --arachne -g file:///tmp/go-lego-reacto.owl --set-important-relation-parent http://purl.obolibrary.org/obo/LEGOREL_0000000 --port 6800 -f /tmp/blazegraph.jnl --export-folder ~/local/src/git/noctua-models/models/ --golr-labels http://noctua-golr.berkeleybop.org --golr-seed http://golr-aux.geneontology.io/solr --skip-class-id-validation --ontojournal /tmp/blazegraph-go-lego-reacto-neo.jnl Have I mixed up the intended arguments? go-lego-reacto.owl and the ontojournal should be the latest from the ieeus-35-neo-test pipeline.

[goodb]

I am having trouble reproducing this. It seems like something happened in the process of saving the model from the form. Other models I've looked at there and locally seem to be working with the reasoner.

[goodb]

@kltm the minerva config there appears fine to me.

[goodb]

when I look at the referenced model in the form, it is not showing any content.
http://noctua-dev.berkeleybop.org/workbench/noctua-form/?model_id=gomodel:5ea8c44700000037

though there is stuff in it according to the owl view. http://noctua-dev.berkeleybop.org/download/gomodel:5ea8c44700000037/owl

Running the reasoner really should not do anything to the model.

[goodb]

@vanaukenk I'm wondering how the gene [WBbt:0006842] RMG got into that model. I can't seem to find it via the noctua type aheads in the form or graph editor.

So far I can not reproduce the misbehavior here. Was that particular model created from scratch in the form today or is this one that has been around ? Is there a problem with any of the other models ?

[vanaukenk]

@goodb I created that model from scratch in the form this morning. This is what the table view of the form entry looked like after I saved the entry. [WBbt:0006842] RMG was entered as an cell type:

[kltm]

(I'm glad we're testing.) @vanaukenk @goodb There is no evidence that this was actually saved, as it never appeared on disk. In fact, nothing seems to have been "saved" since the 13th. Saving a new model just now, nothing appears on disk. There seems to be something going wrong fundamentally here...

[goodb]

@kltm there are two "save" operations. One writes to the live blazegraph instance. This is clearly working. There may be a problem with the periodic dump to .ttl files. I have never really looked at that part.

thanks @vanaukenk I thought that was a gene term..

[kltm]

@vanaukenk The dev setup has been pretty much wiped and rebuilt, in an attempt to clear things up to help with the debugging. Could you start inputting a similar model again? Also, if we can coordinate, we can try and catch it in the act. As well, if we have a script we could work from, we could maybe remove you from the loop... But first, it might be good to see if this was just some crufty "whoops" (we've done a bunch of restarts and tests recently and I may have messed something up), or if there is something actually happening.

[kltm]

@goodb I think I can reliably replicate this now. You can adapt this to run locally and it only requires minerva (and barista). The current command I am running is:

java -Xmx64G -cp ../minerva/minerva-cli/bin/minerva-cli.jar org.geneontology.minerva.server.StartUpTool --use-golr-url-logging --use-request-logging --arachne -g http://skyhook.berkeleybop.org/issue-35-neo-test/ontology/extensions/go-lego-reacto.owl --set-important-relation-parent http://purl.obolibrary.org/obo/LEGOREL_0000000 --port 6800 -f /tmp/blazegraph.jnl --export-folder ~/local/src/git/noctua-models/models/ --golr-labels http://noctua-golr.berkeleybop.org --golr-seed http://golr-aux.geneontology.io/solr --skip-class-id-validation --ontojournal /tmp/blazegraph-go-lego-reacto-neo.jnl  > /tmp/minerva.log

with dev minerva, noctua-models, and noctua.

get model: http "http://barista-dev.berkeleybop.org/api/minerva_public_dev/m3Batch?token=&intention=query&use-reasoner=false&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22get%22%2C%22arguments%22%3A%7B%22model-id%22%3A%22gomodel%3AR-HSA-1169091%22%7D%7D%5D" > /tmp/get-model-1.json get gpad export: http "http://barista-dev.berkeleybop.org/api/minerva_public_dev/m3Batch?token=&intention=action&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22export-legacy%22%2C%22arguments%22%3A%7B%22format%22%3A%22gpad%22%2C%22model-id%22%3A%22gomodel%3AR-HSA-1169091%22%7D%7D%5D" > /tmp/get-gpad-1.gpad get model w/reasoner: http "http://barista-dev.berkeleybop.org/api/minerva_public_dev/m3Batch?token=&intention=query&use-reasoner=true&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22get%22%2C%22arguments%22%3A%7B%22model-id%22%3A%22gomodel%3AR-HSA-1169091%22%7D%7D%5D" > /tmp/get-model-reasoner-1.json get gpad export: http "http://barista-dev.berkeleybop.org/api/minerva_public_dev/m3Batch?token=&intention=action&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22export-legacy%22%2C%22arguments%22%3A%7B%22format%22%3A%22gpad%22%2C%22model-id%22%3A%22gomodel%3AR-HSA-1169091%22%7D%7D%5D" > /tmp/get-gpad-2.gpad get model w/reasoner: http "http://barista-dev.berkeleybop.org/api/minerva_public_dev/m3Batch?token=&intention=query&use-reasoner=true&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22get%22%2C%22arguments%22%3A%7B%22model-id%22%3A%22gomodel%3AR-HSA-1169091%22%7D%7D%5D" > /tmp/get-model-reasoner-2.json get model: http "http://barista-dev.berkeleybop.org/api/minerva_public_dev/m3Batch?token=&intention=query&use-reasoner=false&requests=%5B%7B%22entity%22%3A%22model%22%2C%22operation%22%3A%22get%22%2C%22arguments%22%3A%7B%22model-id%22%3A%22gomodel%3AR-HSA-1169091%22%7D%7D%5D" > /tmp/get-model-2.json

Note the degradation of the responses:

ls -AlFrt
-rw-r--r-- 1 sjcarbon sjcarbon     126020 May  1 17:41  get-model-1.json
-rw-r--r-- 1 sjcarbon sjcarbon       4891 May  1 17:42  get-gpad-1.gpad
-rw-r--r-- 1 sjcarbon sjcarbon     128101 May  1 17:42  get-model-reasoner-1.json
-rw-r--r-- 1 sjcarbon sjcarbon       4891 May  1 17:42  get-gpad-2.gpad
-rw-r--r-- 1 sjcarbon sjcarbon      43132 May  1 17:42  get-model-reasoner-2.json
-rw-r--r-- 1 sjcarbon sjcarbon      43133 May  1 17:43  get-model-2.json

From this state, I do not seem to be able to recover the original model. This is almost certainly not the minimal set to cause the issue, but playing around a little, it seems that turning the reasoner on and off is fine, until the gpad getter is used, then things bad.

[kltm]

As a note to the above, besides the startup warnings, there are no errors produce in log or in to any standard out/error.

[goodb]

Well some progress for hour 1 on this. The first difference between the good and screwed up models is when you see the relations disappearing. In the OWL, the annotations on the axioms that represent these relations are getting dropped. Example

good relation in working model:

[ rdf:type owl:Axiom ;
   owl:annotatedSource <http://model.geneontology.org/5ead99b800000000/5ead99b800000002> ;
   owl:annotatedProperty <http://purl.obolibrary.org/obo/BFO_0000050> ;
   owl:annotatedTarget <http://model.geneontology.org/5ead99b800000000/5ead99b800000003> ;
   <http://geneontology.org/lego/evidence> <http://model.geneontology.org/5ead99b800000000/5ead99b800000007> ;
   <http://purl.org/dc/elements/1.1/contributor> "http://orcid.org/0000-0002-7334-7852"^^xsd:string ;
   <http://purl.org/dc/elements/1.1/date> "2020-05-02"^^xsd:string ;
   <http://purl.org/pav/providedBy> "http://geneontology.org"^^xsd:string
 ] .

degraded relation in borked model:

[ rdf:type owl:Axiom ;
   owl:annotatedSource <http://model.geneontology.org/5ead95c200000000/5ead95c200000002> ;
   owl:annotatedProperty <http://purl.obolibrary.org/obo/BFO_0000050> ;
   owl:annotatedTarget <http://model.geneontology.org/5ead95c200000000/5ead95c200000003> ;
   <http://geneontology.org/lego/evidence> <http://model.geneontology.org/5ead95c200000000/5ead95c200000006>
 ] .

So although the noctua client is reporting no relations, they are actually there, but missing annotations. Bug or feature for Noctua?? Anyway, certainly minerva bug. Now to find out where that is happening..

[goodb]

Doesn't seem to have anything to do with the reasoner or gpad stuff. Something with save. To reproduce

1) create new model with an edge. 2) give it a name and save it 3) turn off the server 4) launch the server and look at the model the edge is missing.

[goodb]

Fun new fact. This only happens when the model is first created. If the same model is loaded again after the relaunch and then edited, everything seems to work, even after subsequent relaunches.

[goodb]

Funner new fact. The missing relations are actually in the blazegraph journal. It seems there is a problem reading them back into the OWL API. They are getting misconstrued as OWLAnnotationAssertions instead of OWLObjectPropertyAssertions. e.g.

AnnotationAssertion(<http://purl.obolibrary.org/obo/RO_0002333> <http://model.geneontology.org/5eadc46a00000003/5eadc46a00000005> <http://model.geneontology.org/5eadc46a00000003/5eadc46a00000004>) 

Maybe linking back to Dustin's missing relations... If it had the RO definitions for these things this probably wouldn't happen.

[balhoff]

@goodb in OWL stored in RDF, property types must be declared, either in the ontology or in an import. Usually when models are read out of the triplestore, go-lego is already loaded in the same OWLOntologyManager. Did this change?

[goodb]

It seems that there is nothing in the OWLOntologyManager when it is loading the models.

There is a method in CoreMolecularModelManager line 1316

static OWLOntology loadOntologyDocumentSource(final OWLOntologyDocumentSource source, boolean minimal, OWLOntologyManager manager)

that is called with the 'minimal' parameter set to true for two loadModel methods in BlazegraphMolecularModelManager. That results in models being loaded with an empty manager.

flipping that to true is looking like a fix to this problem.
Not sure why when or if that was changed...

[goodb]

PR from yesterday https://github.com/geneontology/minerva/pull/316 doesn't fix the original reason for this ticket after all. Without an intermediate restart, still appears to be dropping individuals. Similar in behavior to https://github.com/geneontology/minerva/issues/249 but without any error messages reported by the server. I am able to recover the model by exporting to OWL and then reloading the graph editor.

It feels like a problem with saving and synchronization of the hashmap used to keep the models in memory on the server with the blazegraph store.

[vanaukenk]

@goodb @balhoff @kltm I was following the thread over the weekend. Let me know if/when there's anything more you'd like me to test. Thx.

[kltm]

@vanaukenk The new code is up and the results from my basic test in https://github.com/geneontology/noctua/issues/670#issuecomment-622634463 show no obvious issues. I believe it's ready to test. Be merciless.

[vanaukenk]

@kltm @goodb @balhoff

So far, so good this morning with testing a new model.

I've entered new annotations, imported existing annotations, cloned evidence, created causal relations, run the reasoner, fixed validation errors, edited content in the form table, and checked the annotation preview.

Everything, so far, is working as expected.

Thx.

[kltm]

@vanaukenk I've made some adjustments to the running config and restarted the services, returning noctua-dev minerva to a "managed" state (such as it is). There should be no difference, but I'd appreciate one last glance. Otherwise, once you're satisfied that there are no more serious issues, feel free to close this out.

[vanaukenk]

Closing this ticket for now. The issues appears to have been resolved.