Fix FBTests on JSD2 branch

[fbugissues]

Originally reported on Google Code with ID 7166

All automated tests (FBTests) should pass on JSD2 branch.

This report can (will probably) have follow up reports, but for now we need at least
a branch where to start coding.

Honza 

Reported by odvarko on 2014-02-05 16:07:20

[fbugissues]

Why a branch instead of committing directly to jsd2?

https://etherpad.mozilla.org/firebug-fbtest has details on individual broken tests.

Reported by simon.lindholm10 on 2014-02-05 17:40:42

[fbugissues]

> Why a branch instead of committing directly to jsd2?
Ok, I'll commit directly to JSD2 (and use this place for comments)

Honza

Reported by odvarko on 2014-02-06 07:45:08

[fbugissues]

Agree, we don't need a new branch for this.
We may want to turn this issue into a meta-issue and create a blocking issue for each
failing test. Or do you think that's too much effort?

Sebastian

Reported by sebastianzartner on 2014-02-06 07:46:47

[fbugissues]

I think it is only useful for complex fixes to bring.

Florent

Reported by florent.fayolle69 on 2014-02-06 07:57:11

[fbugissues]

Here is the strategy I have in mind:

1) I believe that it wouldn't be effective to do an issue report for every failing
test (at least at this moment). Many tests fail because of the same reason (e.g. an
existing FBTest API isn't async, we have CM now, etc.) and we would have a lot of duplicities.

2) We should first identify these core reasons and fix them step by step. Step by step
can also mean "group by group" since tests within the same group mostly share the same
skeleton/FBTest API. E.g. if you take a look at Net panel tests, they are mostly the
same (similarly for console/api or commandline/api tests).

3) I want to start with the "examples" group. This group shows how other tests should
be implemented (how the "skeletons" should look like and work). As soon as this group
work, we'll see how to adopt tests in other groups (we might want to also append new
examples).

4) We can call this issue meta, but I'd like to go ahead and start fixing the core
problems as part of this issue - not necessarily create an issue report for every little
thing. But we can create new follow-ups for complex problems (e.g. refactoring MutationRecognizer
in case we would decide to do so).

5) So, I expects follow-ups at the moment when we know what is causing failures and
it's not trivial to fix it - i.e. we have clearly defined task. Note that I am thinking
about FBTest API related problems, not about individual tests related bugs. In other
words, this report and follow-ups should focus on fixing tests that were already working
before.

---

Since fixing all tests is a big task, we should all participate (even of I am assigned
to this report). So, if you have done some analyses of the current test failures (on
jsd2 branch) and you figured out what's causing them, let's discuss it here and:
(a) create follow-up report that can be fixed as separate task (if it's complex fix
or if it needs more discussion)
(b) fix it as part of this bug if it's trivial

Honza

Reported by odvarko on 2014-02-06 08:52:59

[fbugissues]

First commits at:
https://github.com/firebug/firebug/commit/992677eb62e1678f97305e28047eeabd424f4eda
https://github.com/firebug/firebug/commit/2cf1f462fefabfe941fdd4383cea9f541b6d1bbb

1) There is a new Firebug.dispatchEvent() method firing some events to parent TabBrowser
element. These events are utilized by FBTest to recognize some internal state changes
that are not observable through MutationRecognizer (e.g. tab-attached, thread-attached).
It's perhaps not ideal, but surprisingly simple and safe.

2) FBTest.openFirebug() is asynchronous since Firebug needs to attach to the backend
now (over RDP) before executing next test steps.

3) The "examples" test group passes for me. Test drivers are adopted using openFirebug
asynchronously.

Does it pass for you on your machines?

---

I am now focusing on Script related tests.

Honza

Reported by odvarko on 2014-02-07 08:14:57

• Status changed: Started

[fbugissues]

Works fine for me on my Win7 PC. Good job!

Sebastian

Reported by sebastianzartner on 2014-02-07 09:19:03

[fbugissues]

> Works fine for me on my Win7 PC. Good job!
Thanks. Can anyone try it too, I'd like to make sure
it's solid from the beginning.

Honza

Reported by odvarko on 2014-02-07 10:07:59

[fbugissues]

Btw. I just ran the test cases several times. I didn't have time yet to check the code,
though.

Sebastian

Reported by sebastianzartner on 2014-02-07 11:12:31

[fbugissues]

> Does it pass for you on your machines?

I'll take a look this week-end on my Linux machine ;).

@Honza: Do you have a Mac? Or should we CC some Mac guys (Thomas and/or Steven).

Florent

Reported by florent.fayolle69 on 2014-02-07 13:02:45

[fbugissues]

> @Honza: Do you have a Mac?
Only VM

> Or should we CC some Mac guys (Thomas and/or Steven).
Good point. Steven, Thomas, can you please run the tests on your machines
and make a comment about how it works for you? Thanks!

---

Further progress:

1) New method: FBTest.enablePanels(list-of-panels, callback)
- enables given panels (an array) one by one (asynchronously) and executes the callback
after it's done.

https://github.com/firebug/firebug/commit/372824f04b16ebe33a87ceee5d8cf4c3f5200844

2) Script/breakpoints tests pass for me.
(please check on your machines)

https://github.com/firebug/firebug/commit/55e41e12f31f15c2f28facc3e8ac3aa71064cb4f

Honza

Reported by odvarko on 2014-02-07 13:55:58

[fbugissues]

Test result for OSX 10.9.1, jsd2, head@55e41e1

Reported by thomasmrandersen on 2014-02-07 15:25:29

---

• Attachment: fb-test-result-jsd2-2014-02-07.txt

[fbugissues]

Thanks Thomas!

For now it's ok to run only following groups:
* examples
* script/breakpoints

(we are still working on fixing the others)

So, I can see that:
script/breakpoints/4889/issue5027.js

... is failing for you, which should not at this point
(I think I know what the problem is, I'll fix it).

Honza

Reported by odvarko on 2014-02-07 15:33:42

[fbugissues]

Thomas, can you please send me tracing log (*.ftl) produced when script/breakpoints/4889/issue5027.js
test fails? I need to see:
ERRORS, CONNECTION, TESTCASE logs
(note that CONNECTION option needs browser restart)

Honza

Reported by odvarko on 2014-02-08 08:00:20

[fbugissues]

Adopting script/callstack tests
1) script/callstack/5544 fails because of issue 7212
2) script/callstack/3645 and script/callstack/4845 disabled for now (due to issue 6907)

https://github.com/firebug/firebug/commit/be2bec05bc860ab0b882e79561f6bf953f1da131

Honza

Reported by odvarko on 2014-02-08 08:14:39

[fbugissues]

Hi,
Here is the trace log for 4889
note: as you probably see the log also contains some entries from before the test was
run

Reported by thomasmrandersen on 2014-02-08 11:59:48

---

• Attachment: breakpoint-issue5027.ftl

[fbugissues]

> * examples
> * script/breakpoints
Those are working for me.

Reported by simon.lindholm10 on 2014-02-08 19:26:40

[fbugissues]

script/breakpoints/4889/issue5027 works sometimes. I have not found a pattern yet.

Reported by thomasmrandersen on 2014-02-08 19:32:57

[fbugissues]

a bit unrelated, but I am not able to open the trace console if the test console has
not been opened yet/first.

Error message: "FBL is not defined"

Anyone else who has this issue?

Reported by thomasmrandersen on 2014-02-08 19:45:01

[fbugissues]

My results Honza (based on the latest commit of the JSD2 branch) :
https://getfirebug.com/testresults?userheaderid=8f9d0e1648c898afa0ee9cc485000bbd

Florent

Reported by florent.fayolle69 on 2014-02-09 22:59:12

[fbugissues]

Thomas, Florent, make sure you are running FBTest and Firebug both from the JSD2 branch.
Florent's results (comment #20) says FBTest 1.12b4 and Firebug 1.13.jsd2.0a8

The log attached in comment #16 shows weird exceptions. Also, Thomas make sure to disable
EventBug and any other extensions.

Thanks!
Honza

Reported by odvarko on 2014-02-10 07:03:24

[fbugissues]

I'll check that again but I thought FBTest pointed to the versionned folder.

Florent

Reported by florent.fayolle69 on 2014-02-10 07:12:37

[fbugissues]

Here is the trace log again (ref. comment #16)

Same setup, but with a clean browser profile

- Firefox 27
- Firebug is pointing to the repo.
- FBTest, FBTrace built from jsd2 branch.
- FBTrace runs with ERRORS, CONNECTION, TESTCASE checked

The test seems to always pass now. I've experienced some unstabilities with this earlier.

Reported by thomasmrandersen on 2014-02-10 08:00:04

---

• Attachment: breakpoint-issue5027-test2.ftl

[fbugissues]

Test results for Examples and Scripts/breakpoints (ref. comment #12) with a clean browser
profile / jsd2, head@3ad5ecc

https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=d4da5c1d94e17d994bd51f6010000c8f

Reported by thomasmrandersen on 2014-02-10 08:11:43

[fbugissues]

> Florent's results (comment #20) says FBTest 1.12b4 and Firebug 1.13.jsd2.0a8
These *are* the latest versions. Honza, you didn't make an 1.13 release of FBTest so
far. So I guess it's time to do that now.

Sebastian

Reported by sebastianzartner on 2014-02-10 08:26:09

[fbugissues]

Same for FBTrace, btw.

Sebastian

Reported by sebastianzartner on 2014-02-10 08:27:40

[fbugissues]

Here are my test results:
https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=59e983ef4d845523824f39348c0006d1

Sebastian

Reported by sebastianzartner on 2014-02-10 09:49:43

[fbugissues]

> Honza, you didn't make an 1.13 release of FBTest so far. 
Everyone should run FBTest from the source, it's safer. There are changes happening
in FBTest (made as part of this bug).

Honza

Reported by odvarko on 2014-02-10 11:35:53

[fbugissues]

>> Honza, you didn't make an 1.13 release of FBTest so far. 
> Everyone should run FBTest from the source, it's safer.
I do. And I far as I know the others do the same. Though this doesn't mean not to make
any releases for these tools anymore. So I did that now.

Sebastian

Reported by sebastianzartner on 2014-02-10 13:34:17

[fbugissues]

> I do. And I far as I know the others do the same.
Well not yet (as far as I can see from the uploaded test results)

> Though this doesn't mean not to make any releases for these tools anymore.
> So I did that now.
Sure we can. But just to make this clear:

1) The FBTest release done by Sebastian can't be used for testing changes in this bug.

2) Everyone should run FBTest from the source - i.e. from the JSD2 branch where changes
are committed.

Honza

Reported by odvarko on 2014-02-10 13:46:15

[fbugissues]

When submitting test results, you should see "jsd2" in version strings e.g. FBTest 1.13.jsd2b1,
Firebug 1.13.jsd2.0a9 (in such case both extensions Firebug and FBTest comes from jsd2
branch). 

I did one test result log here:
https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=c4845d1eedff0192f7d412aa0c0002d0

When opening FBTest console, you can also see the version string in window title bar.

(and, of course, don't forget to pull from jsd2 before testing)

Honza

Original issue reported on code.google.com by odvarko on 2014-02-10 14:21:57

[fbugissues]

>> I do. And I far as I know the others do the same.
> Well not yet (as far as I can see from the uploaded test results)
Everyone of the team, I meant.

>1) The FBTest release done by Sebastian can't be used for testing changes in this
bug. 
> 2) Everyone should run FBTest from the source - i.e. from the JSD2 branch where changes
are committed.
Right. Just to make it even clearer :-) :
The releases I did are based on 'master', not 'jsd2'.

Sebastian

Original issue reported on code.google.com by sebastianzartner on 2014-02-10 14:23:17

[fbugissues]

Other test results (from today):
https://getfirebug.com/testresults?userheaderid=b0e5c800c2c49288d4435af343000470

Note that there are FBTests marked as "Test-pass" but that do not work, for diverse
reasons. So there must be more than 80 failures currently.

Florent

Original issue reported on code.google.com by florent.fayolle69 on 2014-02-10 22:07:11

[fbugissues]

I've noticed that FBTest.clearAndTypeCommand() (and probably FBTest.typeCommand()) was
buggy.

For example, just run that in the FBTrace Console (choose to evaluate in "Firebug Test
Console"):
FBTestApp.FBTest.clearAndTypeCommand('var a = "no selection";var b = window.a || "selection";console.log(b);',
true);

The result is the following the Command Editor:
"var a = no selection;var b = window.a || selection;console.logb);"

That would fix at least console/executeSelection.js but also I guess many others. I
can take a look at some point, but I would also be glad if anyone fixes that (just
tell me).

Florent

Original issue reported on code.google.com by florent.fayolle69 on 2014-02-10 22:33:15

[fbugissues]

> The result is the following the Command Editor:
> "var a = no selection;var b = window.a || selection;console.logb);"

The bug of course being that there is no quotes and there misses an opening bracket
("(").

Florent

Original issue reported on code.google.com by florent.fayolle69 on 2014-02-10 22:35:32

[fbugissues]

Adjusted the sendChar() function in https://github.com/firebug/firebug/commit/5f410e9309eea8f3cdc23746d16e6999a9d91469
according to http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/EventUtils.js#97,
but it doesn't fix the test case yet.

Sebastian

Original issue reported on code.google.com by sebastianzartner on 2014-02-10 23:39:36

[fbugissues]

I committed more fixes and the following groups pass for me.

* script
* script/breakpoints
* script/callstack
* script/stepping
* script/watch

Please run them on your machines and let me know how it works for you. If you submitting
your results, please run only the Script test groups listed above, so it's easy to
see what failures are related.

Honza

Original issue reported on code.google.com by odvarko on 2014-02-11 12:13:59

[fbugissues]

All tests, 
https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=697164523fb6832fdf2943f0030002d7

Original issue reported on code.google.com by thomasmrandersen on 2014-02-11 13:19:53

[fbugissues]

> All tests
Thanks Thomas. Are you sure you have the latest FBTest version?
I am seeing an exception: FBTest.waitForDisplayedBreakpoint is not a function in your
results. This method has been introduced at:
https://github.com/firebug/firebug/commit/83c45f39a58311978a806f7b1054f8bf6dbee99b

Also, could you please only run and submit tests mentioned in comment #37

Honza

Original issue reported on code.google.com by odvarko on 2014-02-11 13:32:50

[fbugissues]

> Are you sure you have the latest FBTest version?

The extension is pointing to the repo where the branch is jsd2
Latest commit in the tests/FBTest directory:

"FBTest: properly wait for unhighlight (related to issue 7166)"
Revision number: d431717

Original issue reported on code.google.com by thomasmrandersen on 2014-02-11 13:37:53

[fbugissues]

script tests only:

https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=5fec1c679b65ba7ec8519ecfac00088d

Original issue reported on code.google.com by thomasmrandersen on 2014-02-11 13:46:29

[fbugissues]

Here are my results:

https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=243467551fd28c2367c7b1f5330007b0

Sebastian

Original issue reported on code.google.com by sebastianzartner on 2014-02-11 14:29:08

[fbugissues]

> Note that there are FBTests marked as "Test-pass" but that do not work,
> for diverse reasons.
I fixed this at:
https://github.com/firebug/firebug/commit/534a845f00f3c959eb956c4d3df9a881637d63c9

Honza

Original issue reported on code.google.com by odvarko on 2014-02-11 16:26:33

[fbugissues]

Something in Firefox Aurora changed and broke the HTML panel search, which several FBTests
rely on. Reported that in issue 7179.

Sebastian

Original issue reported on code.google.com by sebastianzartner on 2014-02-12 13:40:15

[fbugissues]

More fixes related to the Script groups landed in jsd2. Among other things, I have also
split the test driver API into more modules (they are all located under chrome://fbtest/content/api/*
directory) at:
https://github.com/firebug/firebug/commit/59a19ea3b71e836335c22b4f01e5be25a59d9a87

The remaining problem I see is that a test run can cause the next running test to fail
(async clean-up steps probably run when the next test already started). I am working
on this.

In any case, can you run Script panel groups again and upload some results? :-)

* script
* script/breakpoints
* script/callstack
* script/stepping
* script/watch

Honza

Original issue reported on code.google.com by odvarko on 2014-02-12 18:43:24

[fbugissues]

Latest script panel group, https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=7c9428951a7e21ecf7cd8b3b7100040f

Original issue reported on code.google.com by thomasmrandersen on 2014-02-12 19:19:25

[fbugissues]

> Latest script panel group
Thanks Thomas, if you run the failing tests independently (e.g. by clicking directly
on it), does it pass? (in other words, does it fail only if running together with others?)

Honza

Original issue reported on code.google.com by odvarko on 2014-02-13 07:26:38

[fbugissues]

Hi
A test fails regardless if it is run together with others or independently.

Latest script group test results:

https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=79800d39116435dbdd851f8e5b000578

Original issue reported on code.google.com by thomasmrandersen on 2014-02-13 16:52:30

[fbugissues]

Update from my side (89 errors):
https://getfirebug.com/testresults/?dburi=https://firebug.iriscouch.com/&dbname=firebug4&userheaderid=9deb132c77ffdb1e7ce3225867000938

Florent

Original issue reported on code.google.com by florent.fayolle69 on 2014-02-13 20:43:46

[fbugissues]

FBTest.enableConsolePanel's doc comment claims "Enables the Console panel and reloads
if a callback is specified.", but as far as I can tell that is a lie - it never reloads
the page. Should it, or is the comment plain wrong?

If I want to enable the Console panel and reload, how do I best do that in the new
world?

Original issue reported on code.google.com by simon.lindholm10 on 2014-02-13 21:31:38