webrtc / KITE

KITE is a test engine designed to test WebRTC interoperability across browsers
Apache License 2.0
468 stars 126 forks source link

Exception occured running long JS test #122

Closed scastrec closed 4 years ago

scastrec commented 4 years ago

I'm running test with 1 publisher and ~80 subscribers. To do that, I have 1 client, and running a loop which creates all tabs. At some point (~40eme), there is a crash in KITE but it doesn't stop the test and doesn't generate reports.
That's weird.

 java.util.concurrent.CancellationException
    at java.util.concurrent.FutureTask.report(FutureTask.java:121)
    at java.util.concurrent.FutureTask.get(FutureTask.java:192)
    at org.webrtc.kite.tests.KiteBaseTest.testInParallel(KiteBaseTest.java:273)
    at org.webrtc.kite.tests.KiteJsTest.testInParallel(KiteJsTest.java:112)
    at org.webrtc.kite.tests.KiteBaseTest.execute(KiteBaseTest.java:154)
    at org.webrtc.kite.TestManager.call(TestManager.java:209)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2020-05-13 08:30:39,851 INFO  - execute(DEFAULT) completed.
 2020-05-13 08:30:39,860 INFO  - Test PASSED
 2020-05-13 08:30:39,864 INFO  - Matrix runner at last phase, ending. 
 2020-05-13 08:30:39,871 WARN  - Done! All test cases have passed!
 2020-05-13 08:30:39,871 INFO  - shutdownExecutors() done.
Screen Shot 2020-05-13 at 10 36 33
namvuCosmo commented 4 years ago

image

There this thing called expectedTestDuration in the KITEBaseTest. It is to prevent the test from getting stuck. You might want to include it in the payload and give it another try. It doesn't close down the test since it doesn't close the process that run the js code I think.

image

namvuCosmo commented 4 years ago

Btw I don't think 1 client creating 80 tabs is a good idea, but as long as it works for you I guess 😄

scastrec commented 4 years ago

Looks like the JS code still runs because I have my js console.log. I'll try expectedDuration.

How would you do to run 80 tuples. The first one on a specific publisher URL and the other in subscriber URL?

namvuCosmo commented 4 years ago

Kinda, yeah. That's how I set different steps for caller, who creates the room, and calee, who just joins after room creation, in a call as well. You can use the index to set different script for as many participants as you want.

Besides, we do have load test functionality as well, just not open source. For any kinda large tuple size we would switch to load test mode instead.

scastrec commented 4 years ago

Looks like Java KiteBaseTest has currentIteration and nothing in the JS version of KiteBaseTest.

namvuCosmo commented 4 years ago

Yes, we don't have the bandwidth to develop the JS library as the Java one. Even though I wrote the JS one to mimic the Java and to make KITE run Js code, it's not my forte.

scastrec commented 4 years ago

@namvuCosmo I'll give a try to the load test solution from Cosmo.

namvuCosmo commented 4 years ago

If you want to know more about it, you can contact us from our website https://www.cosmosoftware.io/ :)