search

sq / JSIL

CIL to Javascript Compiler
http://jsil.org/
Other
1.73k stars 242 forks source link

Create automation in-browser tests #524

Open iskiselev opened 10 years ago

iskiselev commented 10 years ago

We need create automation in-browser tests, that could be run automatically on different browsers with help of Sauce labs. It will allow us to be sure, that we haven't break anything in real browser (and create matrix of supported browsers). In future we can add some performance test.

iskiselev commented 10 years ago

Firstly we can easily support comparison tests. We should generate one html page starter fr each test, where we should pass console output of .net app. After it we can compare output JS output with sample in browser and report if we pass test.

kg commented 10 years ago

There's one in the tree, or there used to be, but I killed it because it was impossible to make it reliable. It used the sauce API. Sauce (or selenium in general, most likely) is just too unreliable and too slow for it to ever be worth doing complex tests with.

Maybe we could use it to run the entire normal test suite, though - that might be reasonable.

iskiselev commented 10 years ago

As I understand, you tried to use their Selenium, that allow you control everything from test runner? Could you describe a little bit more what problems have you?

My current thought - is to create some JavaScript test runner, that will accept file with list of all test scripts, their starting point arguments and expected result. For each test, it will create new iframe window, where it will run test (so we will have JSIL initiated from clear point for each test), and will read content of element where we put Console output, checking if it correct. After it it will kill iframe, create new and run next test. When it will be completed - it will notify Sause labs with test results, so that we can check them.

iskiselev commented 10 years ago

I have pushed initial version of my SauceLabs experiments here: 6e73c32befa27d822a946796952b9f86680929c1. To run new test (SauceLabTest.BrowsersTest) please update credentials.cs for right SauceLabs username/accessKey. This test should run all SimpleTestCases on last Chrome/Win8.1 in SauceLabs cloud. Some tests are failed - I have not fixed it, really I'm parse strings incorrect somewhere - but it is good to see how it reports about problems. Wait for a feedback.

iskiselev commented 9 years ago

@kg, could you take a look on it when you'll have time? I'm really need your feedback if it is worth to develop or not.

iskiselev commented 9 years ago

I don't want to create PR at this time as my realization is not mature enough - also I've used NuGet for dependencies (in-app web server) - and would like to discuss if you are OK with it. EDITED: Also, I've not tested how it work in VS2010.

kg commented 9 years ago

Sorry I haven't gotten to this. I think I didn't realize you were waiting for me to review your work.