search

geneontology / pipeline

Declarative pipeline for the Gene Ontology.
https://build.geneontology.org/job/geneontology/job/pipeline/
BSD 3-Clause "New" or "Revised" License
5 stars 5 forks source link

Update to latest version of ODK #136

Closed kltm closed 2 years ago

kltm commented 5 years ago

go-ontology wants to use the latest version of the ODK. Update our images. Need to coordinate with @balhoff https://github.com/geneontology/go-ontology/pull/17956

balhoff commented 5 years ago

We updated to v1.2.18: https://github.com/geneontology/go-ontology/pull/17947

Once pipeline is the same, we would like to use some newer ROBOT features.

balhoff commented 4 years ago

Latest now is v1.2.20.

kltm commented 4 years ago

We spent some time working on this today; @dougli1sqrd @balhoff @matentzn . This was an attempt to bypass an ontology issue that halted the pipeline for a couple of days.

1.2.21 seemed to start thrashing and making no progress with

java -Xmx48G -jar /tools/robot.jar --catalog catalog-v001.xml merge -i extensions/go-lego-edit.ofn --collapse-import-closure true --include-annotations false reason -r ELK --exclude-duplicate-axioms true --remove-redundant-subclass-axioms true relax reduce annotate -O http://purl.obolibrary.org/obo/go/extensions/go-lego.owl -V http://purl.obolibrary.org/obo/go/releases/2020-01-23/extensions/go-lego.owl -o extensions/go-lego.owl

Recommended next step is to split the command. From @matentzn:

The command needs to be split into two. jar the problem is likely in this portion of the command: reason -r ELK --exclude-duplicate-axioms true --remove-redundant-subclass-axioms true I would start by removing the two parameters two see whether the issue is caused by one of them. Second step is split the single robot chain into two; maybe that helps. If 1 works, maybe you should run this by Becky; she might know. I can also imagine that reduce defaults to hermit, so that would definetly explain the memory (add reduce -r ELK, but check robot docs for correct syntax)

balhoff commented 4 years ago

Update on the reduce issue: I added a performance improvement to ROBOT. ROBOT just has a release in the last couple days. We need to wait for @matentzn to update ODK with the new ROBOT. Then we can update ODK (and I need to then tweak a command line in the ontology build).

kltm commented 4 years ago

I tried 1.2.22, but still could not pass this step:

[2020-03-06T21:05:33.090Z] ROBOT_JAVA_ARGS=-Xmx48G robot --catalog catalog-v001.xml merge -i extensions/go-lego-edit.ofn --collapse-import-closure true --include-annotations false reason -r ELK --exclude-duplicate-axioms true --remove-redundant-subclass-axioms true relax reduce annotate -O http://purl.obolibrary.org/obo/go/extensions/go-lego.owl -V http://purl.obolibrary.org/obo/go/releases/2020-03-06/extensions/go-lego.owl -o extensions/go-lego.owl

[killed after a few hours; usually 45m]

We need a plan to move forward: we currently cannot release on the old version of the ODK due to a change in the options used in go-ontology (https://github.com/geneontology/go-ontology/pull/19016/files ?) and we cannot run on the new ODK due to the issue above.

Tagging @balhoff @matentzn

balhoff commented 4 years ago

It looks like this was already updated to 1.2.22, so we could close this. But now I want to update to 1.2.23! :-) See https://github.com/geneontology/go-ontology/pull/19864

kltm commented 4 years ago

@balhoff Okay, so the pipeline is 1.2.22 https://github.com/geneontology/pipeline/blob/3c7744fce45018097deaf95c33153a1776087237/Jenkinsfile#L379, but now we need to be at 1.2.23? What issue/bug was this connected to? If incompatible changes were merged into go-ontology master, that could be an issue (although it looks like only a version bump there as well).

balhoff commented 4 years ago

We haven't used any new features. v1.2.23 includes a newer ROBOT with a major performance improvement for robot query which we want to have for the ontology Travis check.

balhoff commented 4 years ago

Completion of #161 requires updating ODK to v1.2.23.

kltm commented 4 years ago

Was there not a breakage in 1.2.33 that we were wanting to avoid? I believe that this is the obo-graphs issue? @dougli1sqrd

balhoff commented 4 years ago

This is true—I just wanted to make sure it was known that we have reasons to upgrade, so hopefully the obographs issue can be looked into in the near future.

kltm commented 4 years ago

@dougli1sqrd @balhoff Okay, let's go ahead and bump the version after the September release gets out. Talking with @cmungall , it seems unlikely that we'd actually run into the problem in our case.

kltm commented 4 years ago

There should be no problems moving forward with this.

kltm commented 4 years ago

Can now move to v1.2.23.