search

ISISComputingGroup / IBEX

Top level repository for IBEX stories
5 stars 2 forks source link

Evaluate Test Frameworks for Eclipse 4 UIs #2819

Closed Tom-Willemsen closed 6 years ago

Tom-Willemsen commented 6 years ago

RCPTT does not work in eclipse 4. At a meeting, it was decided to create prototype UI tests in a small number of frameworks to test their suitability.

For each testing framework:

The same tests should be migrated for each framework. The suggested testing frameworks to start with are:

Kevin to check with Tom G about licensing fees for the commercial frameworks.

Acceptance criteria:

kjwoodsISIS commented 6 years ago

I have obtained list prices for Squish. They also offer discounts if we can contribute an article/case-study for their web-site. Tom G is considering these.

kjwoodsISIS commented 6 years ago

Licensing Squish seems to be a real possibility - it would make sense to evaluate it first. FrogLogic offer a free-trial of Squish.

kjwoodsISIS commented 6 years ago

Other tools are noted here: http://www.softwaretestinghelp.com/best-gui-testing-tools/

AdrianPotter commented 6 years ago

Other frameworks

The link Kevin posted above contains a great many potential UI testing frameworks. I've looked through the list and looked slightly deeper into a few. None jumped out at me as being more apt for the project than those tested here.

AutoIt

First impressions are that this won't work for us as a system testing framework. The installation is very easy and I've gone through the getting started guide quite quickly. The big thing to point out is that this is an automation program, not a testing program. You can create scripts for it to run through, but there's no feedback as to whether a test passes, certainly not in an xUnit form consumable by Jenkins. I found a 3rd party script that functioned nominally as a unit test framework but the non-native support does not count in AutoIt's favour.

Similarly, the scripting is not straightforward. We'd require learning a new scripting language as Java and Python are not supported. Further, the recording tool that was available in earlier versions has been removed, making the writing of complex tests potentially very complicated.

Conclusion

Although getting IBEX running with AutoIt is straightforward, it is not a testing framework so it is impossible to define test cases with it. You can get it to perform steps but it cannot verify outcomes. Even if we wrote our own test framework to work with AutoIt, it's unlikely we could get to the same level of functionality that we need and integrating with Jenkins is likely to pose challenges. I recommend not to adopt.

SWTBOT

Setup

I've been working with SWTBOT for a couple of hours now and I haven't managed to get it to launch IBEX E4 successfully. I can get an IBEX window to come up but many of the elements (perspective switcher, dashboard) are missing. I've got a strong suspicion it's something to do with the run configuration but thus far I haven't been able to find a suitable combination combination.

The SWT syntax looks reasonable, being Java. The test recorder is OK but doesn't allow for inspection, unlike RCPTT and Squish, which is particularly unhelpful for writing new tests. As it runs connected to Eclipse, I am doubtful it could do advanced operations like restarting the AUT, and it would be significantly more cumbersome to run external scripts (e.g. restore server state). This poses significant disadvantages when compared to Squish. It's main advantage over Squish seems to be that, in running within the same Eclipse environment, the identification of objects is easier.

Test conversion

Since I failed at the first step, I haven't done this.

Jenkins integration

We use Jenkins as a service on our host machines I believe, which is apparently a problem: https://wiki.eclipse.org/SWTBot/Automate_test_execution#On_Jenkins. As above, I haven't gotten as far as creating a build owing to issues earlier in the evaluation.

Conclusion

Although the test definition and recording in SWTBOT seems good, the fact I can't get it to launch IBEX properly is a major stumbling block. Feasibly if we could work out the problem and launch successfully then this would be a contender. However, there indications that integrating with Jenkins would be difficult, running inside Eclipse would prevent us from testing several workflows (e.g. restarting AUT, running external scripts) and the definition of test conditions seems lacking. That leads me to think that even working, it would be inferior to Squish. My recommendation is not to adopt.

Jubula

Setup

Jubula requires a magic formula to get it working, which is documented here:

http://jmhofer.johoop.de/java/testing/eclipse/2011/05/13/ui-testing-with-eclipse-jubula-preparing-the-test-object-1.html

It is disappointing that it doesn't work right out of the box because it took me a fair time to spot the crucial steps:

The steps above allowed me to launch the AUT from the standalone Jubula program. You must set the AUT type as rcp. Notably I also tried using the embedded Jubula from within the GUI. The embedded installation doesn't come with the rcp_support.zip file so I don't know what the alternative is in that case. There may be a dependency you can add to the project directly.

One big gotcha I found moving back and forth between the standalone Jubula GUI and the embedded version is they each try to use a database in C:\users\[username]\.jubula. However, if you install one version, then the other version can't use the database and you get connection errors. The solution is to delete that folder and let it start from scratch.

I am concerned that Jubula appeals to this single blog series as one of its main sources of documentation, which starts off by saying:

Unfortunately, working with Jubula turns out to be a bit unwieldy at first

My set up time has been about 4 hours to get a working E4 AUT. I expect this to be much easier for other developers now that the trick is known. The changes to E4 would be saved in the ibex_gui repo and so it would just be a case of downloading and installing Jubula

Test migration

While I was waiting for various installs to complete, I tried to read ahead onto test management in Jubula. Jubula takes a functional testing approach to test cases. That is, you define blocks which perform a particular step, and then build up a test out of blocks. I can imagine this might be a good way of doing things from a test engineer's point of view, and encourages natural language naming and reuse of elements. However, it is a brand new paradigm to learn (as opposed to Python scripting) which adds to the difficulty of working on the IBEX project. I'm going to try and replicate the same simple test that I did in the Squish migration "CurrentConfigNotInDeleteConfigDialog" to see whether those misgivings are well founded.

I went through the subsequent blogs about test definition. I didn't find it easy to work with. I defined a test with a single step, but without the ability to properly run the AUT (see below) I couldn't get a feel for whether my first-pass implementation (again, of just a single step from the test case) was close to working as expected.

Setup (reprise)

Although I managed to get Jubula to launch the AUT, it did not register that it had been successfully launched. Having spoken to @Tom-Willemsen, this is the same problem we had with getting RCPTT to work with E4 so is likely to stem from the same source. That issue received significant attention with no solution.

On further inspection, I'm not sure this is the same issue as with RCPTT as I get the same behaviour with E3; Jubula launches the AUT but does not connect. This may be because I've integrated the additional .jar as described above incorrectly, or completely unrelated

Jenkins integration

Conclusion

I've been trying for a day to get Jubula working with IBEX, to no avail. Given what I've read about the way tests are defined and executed, I do not imagine writing tests to be a straightforward process, especially from the standpoint of a developer who is accustomed to writing scripts. As such, I'm going to stop evaluating Jubula with a recommendation not to adopt.

Squish

Run with E4 AUT

Getting Squish running with the E4 prototype was incredibly easy.

Replicate a simple test

I replicated the test "CurrentConfigNotInDeleteConfigDialog".

The total migration time was a little under an hour. This included the time taken to become familiar with the UI, recorder, and language syntax. I expect future migration to be far quicker, particularly when we have standardised routines (e.g. getting current config, which is a bit harder in E4 than E3).

The test I made is a tad unreliable in as much as the "waiting for instrument to update" dialog keeps randomly popping up in the E4 prototype. I imagine given the flexibility of the language we'll be able to find a workaround, or we'll just fix that issue in the GUI. I don't think Squish is going to be any more sensitive than other frameworks to this issue

The full test script is given below. Notably it does not currently execute the standard setup that the RCPTT test does. I expect this to be relatively straightforward in Squish owing to the flexibility of Python. As you can see, the first block is something that will soon be factored out into a separate routine. Looping over table items is much easier here compared with RCPTT. The only thing I find a bit clunky at the moment is how to access UI elements (e.g. ":Delete Configurations_Table"). However, Python again would give us a significant advantage here over RCPTT in that not only could we factor out the process of activating menus, we can also have a library of component titles so that we can avoid some of the earlier fragility when changing naming within the GUI

# -*- coding: utf-8 -*-

def main():
    startApplication("eclipse")

    # Find out the name of the current config. Not as easy as E3
    activateItem(waitForObjectItem(":IBEX_Menu", "Configuration"))
    activateItem(waitForObjectItem(":Configuration_Menu", "Edit Current Configuration..."))
    current_config = waitForObjectExists(":Edit Configuration.Name:_Text").text
    clickButton(waitForObject(":Edit Configuration.Cancel_Button"))

    # Verify current config isn't in the list of existing configs
    activateItem(waitForObjectItem(":IBEX_Menu", "Configuration"))
    activateItem(waitForObjectItem(":Configuration_Menu", "Configurations"))
    activateItem(waitForObjectItem(":Configurations_Menu", "Delete"))
    for row in waitForObjectExists(":Delete Configurations_Table").items:
        test.verify(row.text != current_config)

Replicate a more complicated test

I replicated OnlyServerDeviceScreensArePreservedOnRestart.test. It's tricky because it requires starting up a 2nd AUT. In Squish that turns out to be very easy and is aided by good documentation. My new test case was

# -*- coding: utf-8 -*-

def main():
    original_client = startApplication("eclipse")
    setApplicationContext(original_client)

    local_screen = "Local_screen"
    remote_screen = "Remote_screen"

    openDeviceScreensPerspective()

    # Add screens
    addDeviceScreen(local_screen, False)
    addDeviceScreen(remote_screen, True)

    # Start a 2nd client
    new_client = startApplication("eclipse")
    setApplicationContext(new_client)

    openDeviceScreensPerspective()

    # Check devices
    table = waitForObjectExists(":Device screens_Table")
    device_names = [table.getItem(i).getData().name 
                    for i in range(waitForObjectExists(":Device screens_Table").getItemCount())]
    test.log("Device names", str(device_names))
    test.verify(local_screen not in device_names)
    test.verify(remote_screen in device_names)

def openDeviceScreensPerspective():
    mouseClick(waitForObject(":Device Screens_ToolItem"), 89, 34, 0, Button.Button1)

def openEditDeviceScreensDialog():
    clickButton(waitForObject(":Device screens.Edit Device Screens_Button"))

def addDeviceScreen(name, remote=False):
    openEditDeviceScreensDialog()
    clickButton(waitForObject(":Device Screens.Add_Button"))
    doubleClick(waitForObject(":Target.Name_Text"), 86, 7, 0, Button.Button1)
    type(waitForObject(":Target.Name_Text"), name)
    mouseClick(waitForObjectItem(":Target_Combo", "Analyser"), 0, 0, 0, Button.NoButton)
    if remote:
        clickButton(waitForObject(":Target.Save this device screen_Button"))
    clickButton(waitForObject(":Configure Device Screens.OK_Button"))

It again took about an hour but there was a significant delay owing to some of the gotchas below. With time I think writing tests should be really quite quick.

Gotchas

Jenkins integration

I've created a test Jenkins build for some Squish smoke tests. The integration itself is fairly straightforward. One significant issue is that there doesn't seem to be a good way to run the tests in the background. That is, Jenkins cannot be running as a service and there needs to be a display connected. This wasn't the case with RCPTT. The advantage of running in this manner though is we can get screenshots at the point of test failure which will help with debugging.

This page has some useful information on automating with Windows:

https://kb.froglogic.com/display/KB/Automation+on+Windows

I've also emailed FrogLogic to see if they have any advice.

The best way I've found that might work for us so far is to leave a remote desktop connection logged on from a server machine. That is, I opened a remote desktop session on my old machine and create a second remote desktop session onto the build machine. Then I closed the remote desktop session to my old machine, leaving it connected to the build machine. The only change I had to make was to disable the remote desktop optimisation as detailed on the page above. This system could quite easily be replicated by leaving a remote desktop connection open from one of our other build servers.

Notably I contacted FrogLogic about the issue. Their response was:

What you mean is a limitation of Windows that affects every automation tool.

The recommend (and only known to work setup) is to be logged in on the computer and avoid RDP. If RDP has been used, one can recover by a) logging on on the "console" of the computer, or via (Tight)VNC which must then be running on that computer, installed as a service (to be able to automate the Windows logon screen).

We have captured our knowledge about this here:

https://kb.froglogic.com/display/KB/Automation+on+Windows

Having spoken to @FreddieAkeroyd, we may also be able to do this with a server machine in a secure room with autologon setup. This system is already used elsewhere at ISIS. This has obvious security implications but it is considered that if the machine itself is in a secure room then having autologon set up with limited network access doesn't pose an unacceptable risk.

Having spoken to colleagues at Tessella, I'm told that VNC might be a solution. I'm told that if you install VNC on a VM and then access it that way then you can start a display session that isn't closed when you exit VNC. That way the test runner can still access the display without having to leave any processes attached. I haven't tried that solution yet.

Other remarks

The response to the query above came through in about 10 minutes. The documentation of the system online is superb. I have been extremely impressed with my interactions with FrogLogic compared with the barely existent documentation of RCPTT.

Conclusion

Given the experience with other frameworks, Squish is clearly the only feasible framework of those tested. There are a few gotchas above that make this less than the perfect platform for us. Dropping certain keystrokes during typing operations could lead to instability that we will need to find workarounds for. It would also be nice if certain workflows were improved. However, I managed to set up the system and migrate two tests with relatively complex validation to Squish within a couple of hours. My overall feeling is that it's been far easier working with Squish than it ever was working with RCPTT.

The major drawback with Squish is that the Jenkins integration has an issue running without any connected display. We may eventually be able to find a better solution for this but there exists a workable solution at least, which does in one respect come with some advantages over the current system (i.e. screenshots).

Assuming we do not want to fall back to one of our previously discussed options, there are two viable alternatives:

  1. Have UI tests that are not automated, but that we manually trigger periodically
  2. Don't have UI tests

The first option makes very little difference as to whether we go ahead and purchase Squish.

Historically, we have sunk a lot of time into RCPTT, and on a day-to-day basis the benefits have sometimes been dubious. However, it has been instrumental in reducing the time taken to approve the quality of our releases and has been the only available tool for reproducing certain reported bugs. If we were to return to a case of having no automated tests then I think it would be costly in the long run.

My personal feeling is that we should go forward with Squish and automate our tests via Jenkins. There are solutions available to us that will allow us to overcome its few limitations. Experience so far has suggested it is far superior to RCPTT and automated UI testing has significant benefits to the project.

AdrianPotter commented 6 years ago

@Kevin-Woods-Tessella, do you have any suggestions for how this ticket should be reviewed? If the recommendation is accepted, what is the next step?

kjwoodsISIS commented 6 years ago

I think we can do one of two things:

  1. we can ask people to read the findings (documented in this ticket) ahead of next week's stand-down day. We can then have a short discussion and make a decision.
  2. Do the same thing ahead of the normal Sprint Review. In any event, we should reach a decision by the end of the current sprint.

If the recommendation is accepted, we should proceed to purchase. We'll need to do this via Tom, but we already have authorisation in principle to go ahead. FrogLogic are offering us a discount on Squish, provided we provide a short article, by way of a reference. STFC policy means this reference cannot be an endorsement of Squish, but we can describe why we chose it and how we are using it.

kjwoodsISIS commented 6 years ago

Having read @AdrianPotter 's analysis of testing frameworks, it does seem that Squish is the stand-out leader. Given the view that it is far superior to RCPTT, what is there to lose by choosing not to use it? I should also say that my previous experience of FrogLogic was that they provide excellent technical support. I am sure they could help us resolve some of the limitations we have encountered.

AdrianPotter commented 6 years ago

I think a short demonstration and discussion at the sprint review would make sense.

By not using it, we'll no longer be able to keep doing automated UI tests when we migrate to E4. We can continue to use RCPTT in E3 if we choose. If we have no automated UI tests then:

One thing I haven't tested with Squish is automated OPI testing. We could not do that in RCPTT owing to limitations in the framework.

AdrianPotter commented 6 years ago

You have to drill down through about 30 layers of object hierarchy but from my limited test you can interact directly with controls in OPIs which is a major potential advantage over RCPTT. It's a little cumbersome but not prohibitive.

AdrianPotter commented 6 years ago

Discussed at sprint review. Consensus was to proceed with purchase