How to - Abort Testing as failure on any single failuire

[PedroMSantosD]

Your question

Hi,

I'm having issues with graphite metrics being populated by metrics of pages where the check is marked as failed for i.e. a prescritpt or a preURL fail, but the check continues: example:

This test was marked as a failure with the following 1 message:

    Could not single click on element with xpath /html/body/bx-site/ng-component/bx-header-ch/div/bx-header-user-menu-ch/div/div[1]/div/wt-lazy/bx-account-menu-navigation/div/section/div/div/div/div[1]/div[2]/button

Is there a way to make Sitespeed abort all tests of all the pages, and not send any information to graphite nor upload the partial results (to my object storage)? that is: on any failuire/error log the error to console and end without further processing? (a config parameter would be nice!)

Reason I need this, is to avoid adding inconsistent metrics and executions annotations to graphite: i.e. if one out of three iterations fail, (timeout on logon box), then I get data for a based-on-two iterations check, instead of the normalised 3 checks the non failing executions produce. It gets reflected on anomalies like shown below

I have checked the documentation for a config parameter to do so, But I have have no luck. Same on past entries of the issues (back to 2021).

Thanks in advance, (running sitespeed 27.6.3, but I don't think it really matters)

[soulgalore]

Hi @PedroMSantosD not today, but maybe we could add something. Personally I think sending the HTML is really useful (so you actually could see what's broken without accessing the log) but aborting sending the metrics maybe would be possible. Not sure exactly how it wold work though. In your example, is it one of X runs that fails or do it fail all the time for all runs?

[PedroMSantosD]

Hi @soulgalore , in my use case, sometimes a preurl fails on one out of three iterations, sometimes on two, sometimes on all of them. Quite random; but the need to prevent metrics remain. (it might make sense to keep the html reports, as you stated for troubleshooting); maybe I can set some sort of global variable on a try/catch?, maybe a variable could be checked to skip publishing to graphite ? Can you point which part of the code would be interesting to 'define' the do not write to graphite (or influx or whichever metrics backend); and where could a variable be set, so that , shall an error (navigation timeout, missed Xpath, etc) triggers, this variable is set to avoid the writing ?

Best,

[soulgalore]

I wonder if there's a way to fix this inside the script? So if you use preURLs etc I would try to move all logic to one scripting file, run the preURL and if it fails, then avoid running the measure command on the URL that you want to measure? Do you think that would be possible?

[PedroMSantosD]

No clues to be honest.

my use cases are

• a preScript, so I can logon (and randomly fails to load the logon Xpath), where it makes sense to stop all processing as logon has failed;
• and preURL (to get cache populated with static contents) and then measure, so again, if cache is not populated the measurement is tainted by static content loaded on the measured page.

In either use case, i would not like to send metrics to backends, but fully fail the test, even if it happens on any iteration of a set of iterations.

a global parameter triggered by the preURL failuire or the preScript setting it, might do the trick, but other than that I'm clueless.

Does it make any sense?

[soulgalore]

I can write pseudo code on how I think it can work. Can you share how you know that the earlier step failed, how do you check that today or how can you check it?

[PedroMSantosD]

Example, on this 'test',

I have the options:

      --preScript
      /tmp/auth/login_mydomain.mjs
      --postScript
      /tmp/auth/logout_mydomain.mjs

the logout step failed, but it kept trying stuff, and at some point; it did its calculations

[2023-09-29 14:22:17] INFO: ... TTFB: 41ms DOMContentLoaded: 99ms firstPaint: 662ms FCP: 662ms LCP: 1.00s Load: 2.62s TBT: 444ms CLS:0.1348

scriptsFailuires.txt

[PedroMSantosD]

I owe you the one with preURL for now... will get back once I find one.

[soulgalore]

Ok, do you have a way to programatically know that the login step failed? Like checking a cookie or what's the easiest way to do that?

[PedroMSantosD]

I can check for the existance of an XPATH, as new 'elements' are added to the page after logon alternatively, the check itself might get a timeout waiting for page load: say:

try {
        // Hit login button
        await commands.click.bySelectorAndWait(context.options.logon.logincssselector);
        // Add text username and password, finding the field by cssselector
        await commands.addText.bySelector(context.options.logon.username, context.options.logon.usernamefieldcssselector);
        await commands.addText.bySelector(context.options.logon.password, context.options.logon.passwordfieldcssselector);
        // click logon and wait
        await commands.click.bySelectorAndWait(context.options.logon.clickbuttoncssselector);
        // wait for the Corresponding selector to be loaded
        await wait.byXpath(xpath, maxTime)
    } catch (e) {
        // Abort all testing !!!
        throw e;
    }

[amitdisha]

hi @PedroMSantosD ,

I will tell you easiest way out . I faced similar situation and below is how i handled it ,

try{ //do you stuff here } catch (e) { commands.screenshot.take('Error'); // capturing screenshot of error await commands.navigate("https://www.sitespeed.io/"); // always navigate to sitespeed.io page in case of error result= 'Login Failed'; context.log.info(e);// printing error on console commands.error(e);//automatic error sending to HTML report commands.error('Login Failed');//custom error sending to HTML report extraMetrics = { Pass: 0,Fail: 1};// sending status as Fail to HTML and db }

Now in grafana , you can filter out data in query (example in influxdb) like where "key": "group" "operator": "!=", "value": "www_sitespeed_io"

[PedroMSantosD]

Thanks. Safe to close now.