hyperskill / hs-test

A framework that simplifies testing educational projects for Hyperskill. It is built on top of JUnit.
31 stars 10 forks source link

Infinite loop creates Cannot check the submission #26

Closed aaaaaa2493 closed 5 years ago

aaaaaa2493 commented 5 years ago

Cannot check the submission. Please send the report to Hyperskill team. stdout:

stderr:

SerVB commented 5 years ago

I've just tried to run the following solution:

public static void main(String[] args) {
    while (true) {}
}

Now the checker just waits for the submission.

There are some methods to set test timeout in JUnit: https://howtodoinjava.com/junit/how-to-force-timeout-in-jnuit-testcase-execution/.

I think we can't just use @Test(timeout = 8000) for start() in BaseStageTest because there are projects where testing is massive and 8 s aren't enough.

What about the last method – assertTimeout? It requires switching to JUnit 5.

We can declare an overridable long getTimeout() { return 8000; } in BaseStageTest and surround String output = run(test); with assertTimeout(ofMillis(getTimeout()), () -> { output = run(test); });. Then we are able to override the time limit for specific tests.

aaaaaa2493 commented 5 years ago

I would like to move to JUnit 5 but I afraid that something can be broken. You can contribute this solution to a separate branch junit5. After I make sure it's all right (doing internal tests) I'll merge it.

aaaaaa2493 commented 5 years ago

Time limits were implemented without moving to JUnit5 in https://github.com/hyperskill/hs-test/commit/e3cc5ff57139c0585de2c9471765a8b2097cc9cf#diff-9ed6dcf96349fa2b4217e07c1314a7c6R184-R224 Each test contains its own time limit, so you can create tests with different time limits. Time limit for the test is 15 seconds by default.