Testing Diversity

[RCL-Carl]

The current testing method using test_server.py and AutoTester covers a lot of ground, which i think is great. But I feel that right now we have placed some artificial limitations. For example, it is my understanding that it is currently not possible to pass arguments to the compilation of the tests when running with test_server.py. This means that the generated js code is limited to whatever defaults and arguments that test_server.py passes to transcrypt when it attempts to compile or run the python code. This means that we can't have a separate test for ecmascript 5 vs ecmascript 6 representations, among other optimization flags. I propose that we add a command line argument to test_server.py that allows the user to augment the flags that are currently passed. This would allow better testing of these features and eventually, integration into travis-ci so that we can have the unit tests run over a diversity of transcrypts configurable options.

Thoughts?

[axgkl]

Hoi Carl,

here a PR which allows to tell the test server which flag sets to test: https://github.com/QQuick/Transcrypt/pull/226

You can supply flags via a 'flags <test_flags, comma sep.ed>' CLI switch or via $TS_TEST_FLAGS. If nothing is given I take the single set -b -n -c -da -e 5, as before.

I did not change the travis default since I have errors here with ./test_server.py 7777 dev flags '-b -n -c -da -e 5, -b -n -c -da -e 6' for the second set.

Note: I don't allow currently to change the flags on the fly via the URL, if you think its useful I can do it without much probs, let me know.

@JdeH, @RCL-Carl : which flag sets should be tested on travis? Currently only this one: -b -n -c -da -e 5

[RCL-Carl]

I'll check it out tomorrow - sounds very promising.

[JdeH]

I think the Travis CI tests started automatically after each commit should use the following switches:

---

Flags that should always be set:

• Complex, debug assert, build: -c -da -b

Flags that should be used in all possible combinations:

• JS-version: either -e 5 or -e 6 (-e 5 is currently the default, but that may change, so best use explicitly)
• Fast calls: -f on or off (less important, but useful if test time doesn't become to long)

---

If Python 3.6 becomes available on Travis, compiling from it yes or no creates even more variants, there are some specific test cases like fstrings. This isn't a switch, Transcrypt just looks at which Python version is used to run the compiler and includes these tests accordingly. So if Python 3.6 is released, the number of testcases will again double. Currently I am testing 3.6 support only locally using 3.6 b 3.

Since at some point we must curb exponential growth of test time, we cannot support testing 'old' JS and Py versions indefinitely for each commit, but lets decide what to drop from the commit test when this becomes a problem. We should keep testing all supported versions before a release, however.

[JdeH]

@AXGKl CI system becoming more powerful all the time! Shall I leave this open until the above switch variations are part of the CI commit test?

[JdeH]

@AXGKl

I've assigned this PRIO 1, since I consider test diversity very important. So 'must' (of course) does not mean that you're under any obligation to do something about it...

[axgkl]

@JdeH

I would immediately add the e 6 tests but they fail currently. Can you confirm or dismiss this? If dismissed I check further why they fail on my machine.
Or should I just commit it and we have travis fails intermittently until someone fixes it ?

[axgkl]

I mean normally I would say adding w/o a doubt but I'm not 100% sure if the fails are not an artifact of the CI test procedure, thats why I'm asking if you see the errors as well. If you say, yes error as well on your machine I'll commit the checks to travis routine.

[axgkl]

@RCL-Carl, @JdeH

Argh the test flag stuff has a huge bug, pls spare your time testing it. Getting back to you when fixed!!

[axgkl]

hi now the flags should work. https://github.com/QQuick/Transcrypt/pull/230

[JdeH]

I've broken itertools for -e 6. Hope it didn't hinder you. About to commit fix.

[JdeH]

Itertools committed. But still a small problem, see #231, Think that -e 6 is missing in the run (rather than compile) mode.

Switches have to be -r -e 6 (since HTML has to be generated for also for the test cases that are unique for es6.)

Note that also some extra testlets are done for -e 6. So probably autotest.py should be used (doing them conditionally), rather than a static list of testlets.

[axgkl]

@RCL-Carl :

whew, this one works now, required a new firefox on travis. https://github.com/QQuick/Transcrypt/pull/233

you can try just overwriting the test_server.py until merged (think @JdeH is already enjoying his totally unearned sleep ;-) )

[RCL-Carl]

Ya - I've tested with the latest and just hacking the -e 6 or -e 5 as necessary and it works great. I think this will be an excellent addition for testing. 👍

[axgkl]

I hope someday, someone (incl. me) will find time to create a real nice test gui.. where you can choose the flags and also test code on the fly and stuff. Maybe an ipython notebook or sth. The current one is really not sth to be proud of, I was just happy to getting sth automated.

[JdeH]

Really very convenient to be able to CI autotest for -e 6 and -e 7 now. Especially since I am dev'ing on Windows. I this way I can immediately see that at least the autotested part hasn't fallen over on Linux. Of course there's the manual testing & eyeballing, but that doesn't have to be done at each push.

[axgkl]

Hi, reopening this, need some questions.

I included now python 3.5, 3.6 and 3.7:

PR: https://github.com/QQuick/Transcrypt/pull/235

On travis you can see the python version if you open the prepare step where I verify whats within the VM.

The testserver also gives at startup the run_transcrypt[py_maj] version it uses, e.g.:

1480417653.93: INFO: TRUNNER : run_transcrypt37

[JdeH]

This all works like charm!