Issue with waitVanish()

[Tset-Noitamotua]

waitVanish() seems to behave not as expected.

I have this Robot Framework testcases:

Wait Vanish
    startApp
    verifyApp
    use keyboard shortcut  \ue022  \ue014  # ALT + F4
    wait vanish     CalcApp.png     timeout=7.7

Wait Vanish Should Fail
    startApp
    verifyApp
    # use keyboard shortcut  \ue022  \ue014  # ALT + F4
    wait vanish     CalcApp.png

Keyword 'wait vanish' is implemented as:

def wait_vanish(self, *args, **kwargs):
        image = args[0]
        self.timeout = kwargs.get('timeout', 5.0)
        self.reg.waitVanish(image, float(self.timeout))

self.reg is defined as self.reg = Region(Screen()) or to be exact I have a global variable s = Screen() and define self.reg = Region(s) in __init__ funciton.

Keyword 'use keyboard shortcut' works and closes the calculator (which is started by 'startApp' keyword) by simulating ALT + F4 keyboard shortcut. It is implemented as:

def use_keyboard_shortcut(self, *args, **kwargs):
        # press and hold modifier key (e.g. ALT, CTRL etc.)
        self.reg.keyDown(args[0])
        # press some regular (non-modifier) key (e.g. 0-9, aA-zZ, DEL, F1-F12 etc.)
        self.reg.type(args[1])
        # release previously pressed modifier key
        self.reg.keyUp(args[0])

Now the problem is that both test cases pass although the second one should fail.

Any idea what´s going wrong here?

[RaiMan]

The only reason I can imagine (without having tested): CalcApp.png cannot be found on the screen at all (with score > 0.7). In this case waitVanish simply returns with false (waitVanish does not throw FindFailed).

You should check this.

[Tset-Noitamotua]

I thought this, too. But when I put click CalcApp.png before wait vanish CalcApp.png in my test case the click works which means that CalcApp.png is found.

[RaiMan]

Ok, I will look into it.

[RaiMan]

checked with the 1.1.0 build from today: works as expected.

Switch on debugging just before the waitVanish in the def wait_vanish Debug.on(3) and directly after that Debug.off()

might be we can see something in the debug output

[Tset-Noitamotua]

It`s amazig! How do you respond so quickly? Do you get notifications on your mobile? :-))))

[RaiMan]

No, by accident. Just sitting in front of my MacBook Air ;-)

... and new-mail-notifications tagged with names like yours I answer as fast as possible ;-)

[Tset-Noitamotua]

Test case 'Wait Vanish' gives me this:

[debug] Region: waiting for CalcApp.png to vanish
[debug] Image: reused: CalcApp.png (file:/C:/SikuliX_POC/robotframework-sikulixlibrary/step_4/library_self_test_images/CalcApp.png)
[debug] Region: checkLastSeen: not there
[debug] Region: CalcApp.png has vanished

Test case 'Wait Vanish Should Fail' gives me this:

[debug] Region: waiting for CalcApp.png to vanish
[debug] Image: reused: CalcApp.png (file:/C:/SikuliX_POC/robotframework-sikulixlibrary/step_4/library_self_test_images/CalcApp.png)
[debug] Region: checkLastSeen: not there
[debug] Region: CalcApp.png has not vanished before timeout

I think the interessting part is [debug] Region: CalcApp.png has not vanished before timeout.

[Tset-Noitamotua]

Another test. I replaced CalcApp.png with string 'SOMETHING' in test case 'Wait Vanish Should Fail'. Test passes, too and gives me this debug result:

[debug] Region: waiting for SOMETHING to vanish
[error] Image: could not be loaded: file:/C:/SikuliX_POC/robotframework-sikulixlibrary/step_4/library_self_test_images/SOMETHING.png
[error] Image: Image not valid, but TextSearch is switched off!

I thing everything is fine but it should just fail and not pass :)

[RaiMan]

Ok, case 1 should return True case 2 should return False case 3 as well

if it is not there from beginning it also returns true.

Just checked: In all cases it returns what it should return: case 1: True (and the case not there from beginning) case 2: False case 3: False

[Tset-Noitamotua]

I forgott to mention

• test case Wait Vanish passes with [debug] CalcApp.png has vanished
• test case Wait Vanish Should Fail passes, too. But with [debug] CalcApp.png has not vanished before timeout

So debug message is ok, but why does my test case Wait Vanish Should Fail not fail? I´m a little 'totally' confused :)

[RaiMan]

I am not the RFW crack, but since your last op in the case is a Click why should it fail? the waitVanish before returns False, but does this matter?

what are the conditions that have to be met, so that a TestCase fails at the RFW level?

[Tset-Noitamotua]

Here I´m back again. Sorry for late reply - but they don´t let me do test automation at work yet, so I have to do it in my spare free time. I wish I could do it every day at work :-/

I´ve found something in RF UserGuide. This is an example of a keyword implementaiton for a custom RF library:

def numbers_should_be_equal(self, first, second):
        if float(first) != float(second):
            raise AssertionError('Given numbers are unequal!')

And here´s how this keyword is used in a RF test:

***Test Cases***
Passing Test Case
    Numbers Should Be Equal    2    2

Failing Test Case
    Numbers Should Be Equal    2    3

Here`s the result of the test execution:

When I replace the line raise AssertionError('Given numbers are unequal!') with return False so that the function looks like this:

def numbers_should_be_equal(self, first, second):
        print '*DEBUG* Got arguments %s and %s' % (first, second)
        if float(first) != float(second):
            #raise AssertionError('Given numbers are unequal!')
            print('FAIL: Given numbers are unequal!')
            return False

Both test caes PASS - even the one which I want to fail:

So I think one need to raise an exception like AssertionError to make a keyword fail. Here is a quote from RF UserGuide:

Keywords report failures with exceptions, log by writing to standard output and can return values using the return statement. ... Reporting keyword status is done simply using exceptions. If an executed method raises an exception, the keyword status is FAIL, and if it returns normally, the status is PASS. - RF UserGuide

At least one keyword of a RF test case must fail to make the whole test case fail.

[RaiMan]

Thanks for the confirmation and pointers. This is what I suspected. (All ??) testing tools work with some interception of assertion errors/exceptions.

keep on the road even if it is not smooth.