ROBOT Testing Matrix - Githubissues

ontodev / robot

ROBOT is an OBO Tool

http://robot.obolibrary.org

BSD 3-Clause "New" or "Revised" License

263 stars 74 forks source link

ROBOT Testing Matrix #369

Open jamesaoverton opened 6 years ago

jamesaoverton commented 6 years ago

Building on the list of projects in #359, it would be very helpful to test ROBOT against a diverse set of ontology projects. We should be able to test a new release candidate against the whole set of projects and see if we've broken anything (before release).

There might be a better name for this, but I think I've heard it called a "test matrix" before. They're most often used to compile and test a single code base on a wide number of platforms. The Rust language has a tool called Crater https://github.com/rust-lang-nursery/crater that they use to test language changes across their entire ecosystem.

The simplest idea I have for this is to just use Jenkins, but I would really really want a declarative configuration for the system. At the other end of the spectrum, we could write a Makefile and some Python.

I'm willing to cover the server to run this.

Thoughts?

cmungall commented 6 years ago

@kltm @dougli1sqrd - Jenkins2?

kltm commented 6 years ago

Jenkins 2 declarative pipeline is lovely, with a couple of quirks. I'm happy to honk on about it if anybody is interested. That said, it's probably not quite a fit to the exact problem that I'm reading, except as an outer-most runner. What would the expected checkable outputs of the system be?

jamesaoverton commented 6 years ago

The goal is to run each project's usual build and test. The output can be pass/fail, but we need to know which version of ROBOT was used, and should be able to dig in and determine whether a change to ROBOT caused the problem and why. A thin layer over Jenkins could work.

I'll take a look at Jenkins 2. Thanks.

jamesaoverton commented 5 years ago

Hey @ontodev/robot-team. With help from @kltm we've got a Jenkins server that watches this repo for Jenkinsfiles and builds JARs for each branch. So now you can easily download the robot.jar for any PR and test it before we merge. Check it out:

https://build.obolibrary.io/job/ontodev/job/robot/

The next thing I want to do is use the latest robot.jar from master to build some ontologies such as OBI, as a further test for ROBOT. I do this already as part of my release checklist, but making it automatic will catch problems sooner, shorten my checklist, and make releases easier.

beckyjackson commented 5 years ago

We now have a PR for an OBI test for ROBOT, only on the master branch: https://github.com/ontodev/robot/pull/580

jamesaoverton commented 5 years ago

Yes, #580 will build and test OBI with the latest ROBOT master whenever the master branch is updated. So if there's some ROBOT change that breaks the OBI build, we should know about it promptly. But we won't bother checking OBI for every change to every ROBOT branch, just master.

We don't want to overdo this by building a pile of big ontologies all the time, but does anybody have suggestions for other ontologies to test? I think we'll add MRO, since it's smallish but logically rich and makes extensive use of ROBOT templates. Becky might want to add some ontologies that she's responsible for.

beckyjackson commented 5 years ago

I think DO would be a good one to add, maybe ECO now that we have the OBI logic and we build that from templates.

cmungall commented 5 years ago

jamesaoverton commented 3 years ago

Jenkins at build.obolibrary.io has been testing OBI against ROBOT master. That server is down right now. But we hit a bug #808 in the 1.8.0 release that passed OBI, and we only found it when MRO builds failed. This points to a need for broader testing.