Testing on Mac

[Inventitech]

This is a a report on the findings of our manual testing session on the Mac (together with @apanichella). We performed two testing scenarios. I am bit unware of how much Mac-specific these problems really are, but I have the feeling they might be present on every arch.

Overall impression

Registration works, Installing works, JUnit listening works (mostly), Transfering intervals works, JSon respects WD Json "format". No apparent performance problems. Awesome :+1:

Scenario 1

Two problems we noted: There were quite a lot of intervals generated in total. I would assume that we see around ~40% more intervals than on Eclipse. One thing that was a bit surprising was that we had one interval typing whose diff cannot be correct because we never did such a big modification. We had only one line change, and this was the interval generated: http://pastebin.com/mhyZBrhL

Grep for "diff" : 5354. One thing that could have happened is that I think it must have confused it with the file that we had previously opened (which was almost empty).

Another thing is the created JUnit interval, which seemed to be one level up in comparison to WD Eclipse. Compare your JUnit interval to the one created by Eclipse: { "je" : { "p" : "83996c0f878837263b72e856de843ede99ec2ee4", "r" : "F", "d" : 0.002000093460083008, "c" : [ { "t" : "1dfdf42503554e7966afb9b914965151fead1d2aTest", "r" : "F", "d" : 0.002000093460083008, "c" : [ { "r" : "F", "d" : 0.002000093460083008 } ] } ] }, "it" : "ju" }

The difference is that for Eclipse, the top-level test case is in the c array, whereas yours has it in a top-level t. Therefore, it seems the structure is one level too high (in comparison to Eclipse). I actually like yours better, but we have to stick with what we have now. Please also test this with more nested-structures. Probably not a big issue.

Scenario 2

In a second testing scenario, we opened IntelliJ, performed a small and short diff on a file, and closed IntelliJ afterwards. These were the transfered intervals: http://pastebin.com/u2qVp9iF As you can see, IntelliJ somehow did not capture the typing interval of the file.

Good Test Scenarios

I have one idea how you could create and validate good test scenarios. Make a simple Java project (one test, one normal class) and create as-similar-as possible Eclipse and IntelliJ projects. Then perform easy and short tasks in both, and compare each of the session's generated intervals (in a local database). For example: (1) Open IDE x/y (2) Close IDE x/y (3) Assertion: Same intervals for x/y? (1) Open IDE x (2) Read one file for ten seconds (3) Close IDE (4) Assertion: Same intervals for x/y? ... and so on

I know this is tedious work, sorry for that :-)

[levaja]

Great, thanks for feedback, I'm looking into it now!

[levaja]

{ "p" : "507d8a10e2cd4bcceac086e32e04737650fbe227", "r" : "F", "d" : 0.004000186920166016, 
    "c" : [ 
        { "t" : "f79494118d375786a3f4927c2c93f21487fa8fa1", "r" : "O", "d" : 0.0010001659393310547, 
        "c" : [ { "m" : "a905598d323b24fbb6747ec7941ac4f86164b966", "r" : "O", "d" : 0.0010001659393310547 } ] }, 

        { "t" : "615dde619390567a5e26cb64d78ac6d7347bdc83Test", "r" : "F", "d" : 0.002000093460083008, 
        "c" : [ { "m" : "827402e191449c9c15c46e556b1d963cf553d2b8", "r" : "F", "d" : 0.0009999275207519531 }, 
            { "m" : "a905598d323b24fbb6747ec7941ac4f86164b966", "r" : "F", "d" : 0.0010001659393310547 } ] } ] }

{ "p" : "fdda4a4a842bd2bb90c6bd3c8076c777264fc204", "r" : "F", "d" : 0.043, 
    "c" : [ 
        { "t" : "615dde619390567a5e26cb64d78ac6d7347bdc83Test", "r" : "F", "d" : 0.014, 
        "c" : [ { "m" : "a905598d323b24fbb6747ec7941ac4f86164b966", "r" : "F", "d" : 0.01 }, 
            { "m" : "827402e191449c9c15c46e556b1d963cf553d2b8", "r" : "F", "d" : 0.004 } ] }, 

        { "t" : "f79494118d375786a3f4927c2c93f21487fa8fa1", "r" : "O", "d" : 0.029, 
        "c" : [ { "m" : "a905598d323b24fbb6747ec7941ac4f86164b966", "r" : "O", "d" : 0.029 } ]  } ] }

I've been examining these test executions (identical tests with identical classes in both IDEs) and here you can see intervals, one from Eclipse and one from IntelliJ, a bit formatted. Technically, they are the same (order of execution is different). Even though I understand what are you saying, at the moment I can't find the difference. I'll look more into this tomorrow.

[Inventitech]

AFAIA, the issue is here (not sure if it also occurs with > 1 interval):

EC: { "je" : { "c" : [ { "t" : "1dfdf42503554e7966afb9b914965151fead1d2aTest", "r" : "F", "d" : 0.002000093460083008, "c" : [ { "r" : "F", "d" : 0.002000093460083008 } ] } ] } }, "it" : "ju" }

However, IJ: { "je" : { "p" : "9708de2538c2f67ed78d64565bc99e69d52e00db", "t" : "adc18e122770f0bb145797627eae43212bd61249Test", "c" : [ ... ] }, "it" : "ju" }

[levaja]

This first interval doesn't look good - no field "p" (for Project Name Hash), no test methods' hashes , etc. Do you maybe have source code that caused this interval?

[Inventitech]

IMHO it is OK. @apanichella, could you post class?

[levaja]

This code block:

        if (test instanceof ITestRunSession) {
            ITestRunSession session = (ITestRunSession) test;
            setProjectNameHash(session.getTestRunName());
        }

implies that every session should have field "p", that's why I don't think this interval is fully correct. And getTestRunName returns "the name of the launch configuration use to run this test" (from documentation), although we should maybe use the name of the project (session.getLaunchedProject().getElementName()) ?

[levaja]

Furthermore, for Scenario 2, it appears that really short typing intervals are not captured in Eclipse either. I'm investigating that now.

[Inventitech]

Hi Igor, no, not every run should have a field p. That's why there is the if :-)

I have the feeling the IntellIJ Junit listener provides us with more info than the Eclipse one. My only concern is that the format should adhere to what we already have. We have a huge analysis pipeline that works on this data, and that I do not want to change at will (even if the IntelliJ reporting is perhaps slightly more logical).

[levaja]

I completely understand your format concern and I agree with you - I am just interested in the scenario that created this interval, so I can adapt IntelliJ intervals accordingly :)

I'm still bug-hunting this phantom typing interval - I've also found some other minor issues, but still trying really hard to figure out why this typing interval, after being saved to MapDB, is lost few second later, when it needs to be transferred.