Concolic tests failing - Githubissues

cs-au-dk / Artemis

automated testing for JavaScript

http://www.brics.dk/artemis/

GNU General Public License v3.0

36 stars 10 forks source link

Concolic tests failing #96

Closed BenSpencer closed 10 years ago

BenSpencer commented 10 years ago

I tried the concolic test suite yesterday (/artemis-code/tests/system/concolic.py) and the following tests are failing for me: boolean-coercion-of-string-to-false, hidden-field, jquery-checkbox-test, jquery-val-test.

The analysis tries to inject new values but the concrete execution does not take the expected path and so does not explore the whole tree.

The problem exists in aa779ecc, so it is not related to the changes to injection and DOM modifications from the past few days, which I initially suspected. Update: I am now suspicious that this was just because I did not rebuild WebKit between these tests?

I know for sure that at least some of these worked when they were added, but I have not had time to recompile WebKit and test those older commits yet.

BenSpencer commented 10 years ago

boolean-coercion-of-string-to-false - This is discussed in #86, we cannot inject the empty string correctly.
hidden-field - This worked at 340153c7, so I will run git-bisect to track it down this afternoon.
jquery-checkbox-test - At 6a2ff63d when this test was introduced it shows no symbolic information. I don't remember where our support for standard checkboxes got to? Would we expect that to cover this case as well?
jquery-val-test - Uses a regex condition, so it was not expected to work when it was created. It seems to remove newlines so maybe this is why it still does not work after Casper added regex support.

BenSpencer commented 10 years ago

The bug with hidden-field.html seems to be introduced between 1da86fcf (good) and 4de6e324 (bad).

Update: it is introduced by a34100a1, but I can't understand why.

BenSpencer commented 10 years ago

boolean-coercion-of-string-to-false - Issue #86.
hidden-field - Fixed in f1a40d5cde557e0fe607f52bf5beee5ac4bac4f5.
jquery-checkbox-test - Bug in the test itself, fixed in aec53ddf8b148c1e635a5cb7d8c51aca0a12abc7 and f1c8852d4c2897d5bbbcdfc219a69759332f638a.
jquery-val-test - Unsupported example, not a bug. Regex support added in 29a0be4b1d67a40dcbe3740f9e0277794fa257fb but now this example causes a type error by having string and integer conditions on a single variable. I have added jquery-string-test which we can solve now.

BenSpencer commented 10 years ago

The only remaining failing test case is now jquery-val-test, which is because of an unimplemented coercion feature (not a bug) and which Casper is already working on, so this issue is closed.