alanruttenberg
commented
4 years ago Ok, some thoughts about this.
1) Make a Github Common Logic "organization", to separate the tests from any specific person or project. Make us members of the organization, and then create repository for tests owned by the organization. Encourage others to submit PRs for new tests. Create other repositories there for other needs (documentation, errata, etc). Pick and choose from that repo which tests to use for HETS builds.
2) Organize the tests along the lines of the OWL2 tests
- Consistency Tests
- Inconsistency Tests
- Entailment Tests
- Non-Entailment Tests
- Syntax Translation Tests (CLIF->CGIF->XCL, maybe abstract syntax, TFOL)
I'd add
- Correct syntax tests
- Incorrect syntax tests (should be rejected by parsers, or warned about if there's a non-strict parsing mode.)
OWL has defined profiles. The analog in would be CLIF semantics versus Full CL. Perhaps your categories FOL,Imp,Seq,Full. Tests would be annotated as to which "profile" they are applicable to.
The OWL tests are mostly meant to stress the reasoner. Here we want to test the language so the emphasis is more on small examples that exercise and illustrate the language, and in particular non-obvious (to various audiences that might be interested) features. Hopefully it will be a relatively light load for the reasoners.
I'm open to suggestions - I'm not saying we necessarily have to do write all the tests ourselves or fiil each of the categories from the start. But set up an inviting infrastructure so there's low friction for others to submit new tests.
If we agree on the general outline we can start discussing the specifics of the format of the tests, and one of us can create the Github organization.
so far, we have some syntax tests that are run (via travis) for every pull request. Probably this needs to be enhanced. Colore could be of use here. Note that we must not integrate too many tests; otherwise travis will take too long.