search

webrtc / KITE

KITE is a test engine designed to test WebRTC interoperability across browsers
Apache License 2.0
468 stars 126 forks source link

JS tests are broken #107

Open ldttmt opened 4 years ago

ldttmt commented 4 years ago

Hi, We are using KITE to test our app with WebRTC but with the recent change our js test is broken. As far as I can tell, it starts from this commit that change the capabilities object: https://github.com/webrtc/KITE/commit/6be3987c88c01e872e107dd40755faf0c1642a6c Are there any other incompatible change with the previous version?

namvuCosmo commented 4 years ago

Hi,

Thanks for reporting. We just update the repository to fix Js test error. The test should run normally now.

Can you give it a try and let us know if there's any issue?

ldttmt commented 4 years ago

Just to clarify, our issue is that the capabilities object structure changed. For example, before to get the browserName in the config, we use capabilities.browserName but now we need to do capabilities.browserSpec.browserName. I looked at the new commits but I don't think there is any changes for backward compatibility.

namvuCosmo commented 4 years ago

It's the kite-common library mostly (0.1.12), it's not backward compatibility per se , but it should work with the new capabilities. I'm sorry for the inconvenience, but I don't have the bandwidth right now to spend more time on Js code at the moment, so maybe if you can bear with me and make the changes to the new structure, or maybe use the older version of KITE.

If you need any help or if there's anything you don't understand, let me know.

The new objects are like this (capabilities object in js code is client object in java code):

image

ldttmt commented 4 years ago

Yes, we have switched to temporarily only clone the older version. I created this ticket to just want to learn if there is any other breaking changes in addition to the one above so we can be aware when we migrate to the new version. By the way maybe next time, you can add gitHub tag when introduce a newer version, so it is easier for us to know when there is a major update. A CHANGELOG.md for the repo would also be great.

namvuCosmo commented 4 years ago

Sure, I'll keep that in mind for next update. Thanks for the suggestion.

Piasy commented 4 years ago

Hi @namvuCosmo , I tried the latest commit, it has two issues:

  1. you commit a test selenium hub url to KITE-AppRTC-Test/configs/js.iceconnection.apprtc.config.json;
  2. after revert this url change, it fails with below logs:
java -Dkite.firefox.profile=/Users/piasy/src/media/KITE/third_party/firefox-h264-profiles/ -cp /Users/piasy/src/media/KITE/KITE-Engine/target/kite-jar-with-dependencies.jar:target/* org.webrtc.kite.Engine configs/js.iceconnection.apprtc.config.json
2020-04-07 19:52:26,956 INFO  - Reading '/Users/piasy/src/media/KITE/KITE-AppRTC-Test/configs/js.iceconnection.apprtc.config.json' ...
 2020-04-07 19:52:27,011 INFO  - new RoomManager(1) for https://appr.tc
 2020-04-07 19:52:27,012 INFO  - Finished reading the configuration file
 2020-04-07 19:52:27,012 INFO  - Creating report folder if not exist at :/Users/piasy/src/media/KITE/KITE-AppRTC-Test/kite-allure-reports/
 2020-04-07 19:52:27,016 INFO  - Running {"type":"interop","name":"IceConnectionTest","testImpl":"IceConnectionTest.js","description":"This test check the ICEConnection state between two browsers communicating via appr.tc","tupleSize":2,"noOfThreads":1,"maxRetryCount":0,"delayForClosing":0,"permute":true} ...
 2020-04-07 19:52:27,030 INFO  - Executing {"type":"interop","name":"IceConnectionTest","testImpl":"IceConnectionTest.js","description":"This test check the ICEConnection state between two browsers communicating via appr.tc","tupleSize":2,"noOfThreads":1,"maxRetryCount":0,"delayForClosing":0,"permute":true} for 1 browser tuples with size :2
 2020-04-07 19:52:27,035 INFO  - current phase in test manager has been set to DEFAULT
 2020-04-07 19:52:27,039 INFO  -  KiteJsTest
 2020-04-07 19:52:27,040 INFO  -
           |--------------------------------------------------
           | Number of registered tests: 0
           | Elapsed time: 0m0s (avg:0m0s)
           | Running test case with ID: 0/1
           | ETA (for IceConnectionTest): Estimating..
           |--------------------------------------------------
 2020-04-07 19:52:27,041 INFO  - execute(DEFAULT) for the test case: MAC_ch-MAC_ch (2020-04-07-195227)
 2020-04-07 19:52:27,042 INFO  - Not load test, init report for test
 2020-04-07 19:52:27,050 INFO  - Finished filling out initial report for phase
 2020-04-07 19:52:27,050 INFO  - payload handling
 2020-04-07 19:52:27,051 INFO  - the payload is {"url":"https://appr.tc","port":30000,"testTimeout":60,"getStats":{"enabled":true,"statsCollectionTime":2,"statsCollectionInterval":1,"peerConnections":["appController.call_.pcClient_.pc_"],"selectedStats":["inbound-rtp","outbound-rtp","candidate-pair"]}}
 2020-04-07 19:52:27,106 INFO  - Creating runners
 2020-04-07 19:52:27,127 INFO  - Starting server on 30000...
 2020-04-07 19:52:27,128 INFO  - Starting the execution of the test runners in parallel
 2020-04-07 19:52:27,129 INFO  - JsTestRunner will execute in js/
 2020-04-07 19:52:27,129 INFO  - JsTestRunner will execute in js/
 2020-04-07 19:52:27,129 INFO  - Executing command: node IceConnectionTest.js 2 0 91eb72e9-ad58-4b08-b39d-c006e981ccd7 ./temp/IceConnectionTest.js_2020-04-07-195227_6d527
 2020-04-07 19:52:27,129 INFO  - Executing command: node IceConnectionTest.js 2 1 91eb72e9-ad58-4b08-b39d-c006e981ccd7 ./temp/IceConnectionTest.js_2020-04-07-195227_6d527
 2020-04-07 19:52:27,529 INFO  - [NodeServer 30000 ] Listening on port 30000
 2020-04-07 19:52:27,726 INFO  - [IceConnectionTest.js_0 ] { WebDriverError: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, goog:chromeOptions: {args: [--use-fake-device-for-media..., --use-fake-ui-for-media-stream, window-size=undefined]}, platform: MAC}
 2020-04-07 19:52:27,726 INFO  - [NodeServer 30000 ] Connected
 2020-04-07 19:52:27,726 INFO  - [IceConnectionTest.js_0 ]     at Object.checkLegacyResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/error.js:585:15)
 2020-04-07 19:52:27,726 INFO  - [IceConnectionTest.js_1 ] { WebDriverError: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, goog:chromeOptions: {args: [--use-fake-device-for-media..., --use-fake-ui-for-media-stream, window-size=undefined]}, platform: MAC}
 2020-04-07 19:52:27,726 INFO  - [IceConnectionTest.js_0 ]     at parseHttpResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:556:13)
 2020-04-07 19:52:27,727 INFO  - [IceConnectionTest.js_1 ]     at Object.checkLegacyResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/error.js:585:15)
 2020-04-07 19:52:27,727 INFO  - [IceConnectionTest.js_0 ]     at Executor.execute (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:491:26)
 2020-04-07 19:52:27,727 INFO  - [IceConnectionTest.js_1 ]     at parseHttpResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:556:13)
 2020-04-07 19:52:27,727 INFO  - [IceConnectionTest.js_0 ]     at process._tickCallback (internal/process/next_tick.js:68:7) name: 'WebDriverError', remoteStacktrace: '' }
 2020-04-07 19:52:27,728 INFO  - [IceConnectionTest.js_1 ]     at Executor.execute (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:491:26)
 2020-04-07 19:52:27,728 INFO  - [IceConnectionTest.js_1 ]     at process._tickCallback (internal/process/next_tick.js:68:7) name: 'WebDriverError', remoteStacktrace: '' }
 2020-04-07 19:52:27,729 INFO  - [IceConnectionTest.js_0 ] wrote all data to file
 2020-04-07 19:52:27,730 INFO  - [NodeServer 30000 ] Connected
 2020-04-07 19:52:27,732 INFO  - [IceConnectionTest.js_1 ] wrote all data to file
 Process exited with result 0 and output { WebDriverError: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, goog:chromeOptions: {args: [--use-fake-device-for-media..., --use-fake-ui-for-media-stream, window-size=undefined]}, platform: MAC}
    at Object.checkLegacyResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/error.js:585:15)
    at parseHttpResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:556:13)
    at Executor.execute (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:491:26)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'WebDriverError', remoteStacktrace: '' }
wrote all data to file

Process exited with result 0 and output { WebDriverError: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, goog:chromeOptions: {args: [--use-fake-device-for-media..., --use-fake-ui-for-media-stream, window-size=undefined]}, platform: MAC}
    at Object.checkLegacyResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/error.js:585:15)
    at parseHttpResponse (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:556:13)
    at Executor.execute (/Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/node_modules/selenium-webdriver/lib/http.js:491:26)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'WebDriverError', remoteStacktrace: '' }
wrote all data to file

2020-04-07 19:52:27,740 INFO  - JsTestRunner filePath = js//temp/IceConnectionTest.js_2020-04-07-195227_6d527/0/result.json
 2020-04-07 19:52:27,740 INFO  - JsTestRunner filePath = js//temp/IceConnectionTest.js_2020-04-07-195227_6d527/1/result.json
 2020-04-07 19:52:27,740 INFO  - resultFile = /Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/temp/IceConnectionTest.js_2020-04-07-195227_6d527/0/result.json
 2020-04-07 19:52:27,740 INFO  - Reading 'js//temp/IceConnectionTest.js_2020-04-07-195227_6d527/0/result.json' ...
 2020-04-07 19:52:27,742 INFO  - resultFile = /Users/piasy/src/media/KITE/KITE-AppRTC-Test/js/temp/IceConnectionTest.js_2020-04-07-195227_6d527/1/result.json
 2020-04-07 19:52:27,743 INFO  - Reading 'js//temp/IceConnectionTest.js_2020-04-07-195227_6d527/1/result.json' ...
 2020-04-07 19:52:27,743 INFO  - Stopping server on 30000...
 2020-04-07 19:52:27,743 INFO  - Server on 30000 thread is completed.
 2020-04-07 19:52:28,746 INFO  - Server on 30000 process stopped.
 2020-04-07 19:52:28,746 INFO  - Server on 30000 stopped.
 2020-04-07 19:52:28,746 INFO  - execute(DEFAULT) completed.
 2020-04-07 19:52:28,749 INFO  - Test PASSED
 2020-04-07 19:52:28,750 INFO  - Matrix runner at last phase, ending.
 2020-04-07 19:52:28,755 WARN  - Done! All test cases have passed!
 2020-04-07 19:52:28,755 INFO  - shutdownExecutors() done.
Charlesjourdain commented 1 year ago

Oui Intent#