search

clulab / processors

Natural Language Processors
https://clulab.github.io/processors/
Apache License 2.0
418 stars 101 forks source link

Unreliable tests #708

Open kwalcock opened 1 year ago

kwalcock commented 1 year ago

For PR #707, the first test run failed at it should "collapse prepositions with verb objects" in org.clulab.processors.TestEnhancedSemanticRoles, even though nothing added there would have affected a test. The second run passed without any changes having been made.

It is known that some Stanford software can get different results on different runs. Is that the case here? The are also testing on Java 11 now, which we're not used to. Strange things could happen.

kwalcock commented 1 year ago

This happened again today.

kwalcock commented 1 year ago

Today TestParallel failed, complaining "should match processing documents serially". This was for the first run of PR #722 (http://jenkins.clulab.org/job/processors/view/change-requests/job/PR-722/1/). A diff shows that enhanced-semantic-roles and semantic-roles are different in the serialized form. So far I can only imagine that happening if the neural networks are getting different answers. The error went away on the second run.