Compilation checks over specifications

[ariannab]

Every specification (Pre, Post, ExceptionalPost) is now checked for compliance before being accepted as correct. If it does compile, the specification is discarded (no translation is produced in output for the respective comment). The compilation issue is logged.

This new feature slightly improves Toradocu's precision.

[ariannab]

According to the log, this pull request fails because of a problem regarding the formatting task. However, I cannot reproduce the problem. The command ./gradlew build -PdisableShadowJar && ./gradlew checkFormat works flawlessly on my machine.

[albertogoffi]

I think the checkFormat task is not the problem here. The build failed because a test failed (https://api.travis-ci.org/v3/job/380762990/log.txt):

* What went wrong:
Execution failed for task ':test'.

The failing test is the following:

org.toradocu.accuracy.AccuracyCommonsCollections4 > testDefaultMapEntry FAILED
    java.lang.AssertionError: @throws recall is different than expected
    Expected: a numeric value within <0.001> of <1.0>
         but: <0.0> differed by <0.999> more than delta <0.001>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.toradocu.testlib.AbstractPrecisionRecallTestSuite.test(AbstractPrecisionRecallTestSuite.java:114)
        at org.toradocu.accuracy.AccuracyCommonsCollections4.testDefaultMapEntry(AccuracyCommonsCollections4.java:101)

[ariannab]

Right, sorry, I misread the log. Anyway, I cannot reproduce the problem. Also, the pull request changes didn't affect that particular test.

[albertogoffi]

Right, sorry, I misread the log. Anyway, I cannot reproduce the problem.

I confirm that all the tests pass on macOS. Instead, there are several failures on Linux (test report attached).

test-report.zip

[ariannab]

That's interesting: it doesn't look like a problem due to the features introduced by the PR. I will attempt a new investigation on a Linux machine.

[ariannab]

@albertogoffi on the Linux machine I tried, even your master branch has failing tests. Could you please try it yourself, just to check?

[albertogoffi]

@albertogoffi on the Linux machine I tried, even your master branch has failing tests. Could you please try it yourself, just to check?

I confirm that. On Linux we have 6 tests failing, even with the master branch version. Interestingly enough, now the 6 tests fail also on my macOS. The good news is that I can reproduce the problem locally.

[ariannab]

Interestingly enough, now the 6 tests fail also on my macOS.

The build works flawlessly for me on macOS (tried on a brand new clean build).

[albertogoffi]

Arianna investigation highlighted a buggy behavior of some Java runtimes. I prepared a package with a minimal failing example. Once extracted run the command: cd toString-bug && javac foo/*.java && java foo.Main The output is printed on the standard output and depends on the Java environment used:

• "java.util.Map$Entry<? extends K, ? extends V> arg0" with jdk 1.8.0_172 (macOS) and openjdk 1.8.0_171 (Linux)
• "java.util.Map.java.util.Map$Entry<? extends K, ? extends V> arg0" with jdk 1.8.0_20 (Linux)

On the Travis side, we could try to fix this using a more recent version of the jdk (https://docs.travis-ci.com/user/languages/java/#Updating-Oracle-JDK-to-a-recent-release). toString-bug.zip

[ariannab]

On the Travis side, we could try to fix this using a more recent version of the jdk

However, since we cannot be sure about the JDK used by the Toradocu users, I could implement a workaround to manage the problematic cases.

[mernst]

I think it's OK to have either a workaround or a run-time test (say, on startup) that falls Toradocu with a clear error message about updating the JDK.

[ariannab]

I would warn the user for sure. Maybe the workaround is not worth it. There are already some bug reports related to the problem:

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8054213

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6376382

[albertogoffi]

I propose to add a message in the README and a check in Toradocu with a specific error message.