Fixed memory leaks that arise while generating solr index entries

[yy20716]

This patch is for solving the memory leak issue that arises while generating solr indexes from go-cam model files.

• Problem: Loading each model file involves (i) retrieving some literal values from ontology objects and (ii) materializing implicit axioms. These literals and axioms are later directly referenced and fed into the object used for solr document objects. The problem is that, even if we put codes that remove ontology objects, garbage collectors in Java skipped releasing the literals and the axioms because they are still in use, therefore all connected objects still stay in memory.

• Solution: I cloned new objects from reasoning and reasoner objects (and then dispose them once they are converted them as solr index entries).

• I believe similar issues would arise for many other parts of owltools but I minimally fixed the codes to avoid any other potential bugs for now.

[lpalbou]

Cool !

[kltm]

Great! What I'm going to do is fix up the dev machine (tomodachi) with the node updates and switch owltools over to this branch for testing.

[kltm]

Okay, so we have a chain of issues going back to how stale tomodachi is. I'll try and clean this up on Monday up the hill.

[kltm]

Updated on server for testing. @yy20716 Could you look into the conflicts with master?

[yy20716]

• I resolved the conflicts as requested. The index dump for solr seems okay from my side, but I really did not have any other ways to confirm them... I hope that everything would work as intended.

• The lessons I learned from this task can be summarized as follows:

  • We need to clone/deep-copy the ontology instance (OWLOntology) first before we run reasoners over the instance. Running reasoner each time inevitably adds many new instances into the ontology instance which cannot be easily cleaned up. This later causes memory leaks. This can be solved by running the reasoner over deep-copied ontology instance and later safely disposing the instance, while we maintain the original one intact.
  • Note that cloning ontology instance cannot be achieved via conventional serializers such as Kryo or Apache Common Lang's SerializationUtils; it should be done via copyOntology method provided by OWLAPI.
  • I added this deep-copying logic in SolrCommandRunner's loadModelAnnotations and loadEachModelAnnotation. Thank you.

[yy20716]

FYI. I just wanted to note that it's not safe to merge this patch yet into the main branch, i.e. this one still uses the codes before this pull request. Once we confirm that the codes in that pull request work as expected, I will then incorporate the codes from that request and make an additional commit into this branch. Thank you for your patience.

[kltm]

I've merged #249 into master.

[kltm]

@yy20716 How is the status now? Tagging #212

[yy20716]

Seth (@kltm), I think it's now good to go but please let me know if you encounter any issues. I only tested this with dummy solr loader, so maybe there are something I couldn't consider before. Thank you for your time.

[kltm]

@yy20716 @cmungall As per our discussion, let's go ahead with this code and test in amigo 2.5.x/master to optimize the time we have with @yy20716 .