test suite: keep going on failure, run --update-check and push results to github

[adriaanm]

if the junit test suite fails, we won't see output from partest

if partest fails, it would be great to re-run it with --update-check --failed and push the changes to some repo so we can analyze them outside of jenkins

anything we can do to make it super easy to take a jenkins job and pick up where it failed on our local machines is worth doing

[lrytz]

speaking of junit vs partest, i've recently added infantile support for run tests in JUnit: https://github.com/scala/scala/pull/5059/files#diff-b5fb27444b0e5fec3e801735ea01e2f9

Here's some pros / cons of moving to JUnit:

pro

• easier to run a specific test
• stay in IDE
• an individual test runs faster
• potentially more flexible, easier to write helpers that are shared across tests

cons / todos

• source code in strings: no ide support for test code itself
• don't have all of partests functionality (yet)
  • java compilation (only mixed, could probably be implemented)
  • --grep (could probably be implemented in junit)
• harder to check compiler / test output (can probably be done)
• no --update-check
• does not run tests in parallel (need to implement, JUnit provides some support it seems, to be seen how hard it is)

[adriaanm]

It would be awesome to standardize on junit. We can probably overcome those cons. We could use quasi quotes for the code. IntelliJ should support that.

[adriaanm]

I had a feeling tests do run in parallel in the ide already?

[retronym]

IntelliJ can be configured to inject Scala syntax into strings. https://www.jetbrains.com/help/idea/2016.1/using-language-injections.html?origin=old_help On Thu, 31 Mar 2016 at 06:12, Adriaan Moors notifications@github.com wrote:

I had a feeling tests do run in parallel in the ide already?

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/scala/scala-jenkins-infra/issues/169#issuecomment-203612931

[lrytz]

cool! though it doesn't seem to mix well with multi-line strings / stripMargin

[retronym]

Probably the same bug as https://youtrack.jetbrains.com/issue/SCL-9033

I'd just get rid of the stripMargin until that is fixed.

[retronym]

BTW, JUnit "run" tests could also load source files as classpath resources. Any String IntelliJ that refers to a path to a classpath resource gets autocompletion, refactoring, find usages, navigate to source support.

[som-snytt]

I'm not up to speed on the issue, but having a single test framework (such as under junit) is a great goal, even if functions like update-check have to be pushed out to a tool.

The possible differentiator may be, "fast" tests are junit, while "slow" tests are partest, where a slow test requires initing a compiler.

[lrytz]

The possible differentiator may be, "fast" tests are junit, while "slow" tests are partest, where a slow test requires initing a compiler.

We already have JUnit tests that instantiate a compiler, for example https://github.com/scala/scala/blob/2.12.x/test/junit/scala/tools/nsc/backend/jvm/opt/InlinerTest.scala. In fact there's a huge performance advantage over partest: a single compiler instance is used for every test in this file, while partest creates a new one for each test.

[SethTisue]

meh, years have passed and we've been okay with the status quo