CI for testing behaviors of Annotators

[hiroshinoji]

There are test classes for some annotators (https://github.com/mynlp/jigg/tree/master/src/test/scala/jigg/pipeline), but they are only limited, or very superficial.

One particular problem is that we cannot test the behavior of annotators that rely on external model files, including CoreNLP annotator, since Jigg does not have the models of them internally. This is also the case for annotators for non-maven softwares, including mecab, KNP, etc.

Currently, for example, when I update the version of CoreNLP in build.sbt, I don't carefully check how the behaviors of annotators change; I just see that Jigg's wrapper does not output errors when executing. This is bad.

We need more systematic test mechanism for these external softwares, perhaps with some CI tool?

[fyamamoto10]

Memo

Triggers of Travis CI:

• push or pull requests
• Cron Jobs (at regural scheduled interval)
  • interval : daily, weekly, monthly

When a programing language is scala:

• If there is build.sbt file in the root repository, uses sbt to build it.
• To get the latest version of extarnal software, write latest.integration in build.sbt
  • For example,

    libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "latest.integration"

    Now(2018/05/23), get the version 3.9.1. If you get the model (English) file of Stanford-CoreNLP,

    libraryDependencies += "edu.stanford.nlp" % "stanford-corenlp" % "latest.integration" classifire "models"

  • Other writing: 3.8.+ or [3.8, )
  • By default, get the latest version software in the Mave Central Repository.

[hiroshinoji]

Now the tests are working (https://travis-ci.org/mynlp/jigg).

We want to add several other tests for performance as well as corner cases. For example, maybe some parser does not accept a very long sentence and possibly throw an error. But I think Jigg does not implement special mechanism to handle such errors.

We should first check the behaviors of the original parsers (corenlp[parse], berkeleyparser, depccg, etc.), and then create tests by defining the desirable Jigg's input and output pairs, which would absorb the errors of the original parsers.