search

salesforce / utam-js-recipes

Examples of testing with UTAM JavaScript
MIT License
48 stars 23 forks source link

[Bug] Incomplete jasmine stack-trace in case of utam method caused a test failure #122

Open tanujvishnoi opened 1 year ago

tanujvishnoi commented 1 year ago

Actual behavior

Hey everyone, we're working with utam-webdriver-jasmine flavors along with TS and during spec run we have noticed the stack trace get truncated if a test failed because of web-element issue [utam waitfor- method or element is not in scope], in-short we got a complete stack trace in case of assertion failure like which spec line and file, but other failures wont populate complete stack-traces & the error source,

What is your expected behavior?

during any failure, the complete stack-trace should be visible indicating the spec line which actually caused this issue

Steps to reproduce

1: Create a TS-Jasmine-utam-wdio based project 2: Create few utam files with wait for methods that will fail or some non-existing locators 3: Draft spec file and use utam file and called specific methods or access locators stated in step2 4: Run the spec and observe the stack traces

Environment

Additional context

Refer to attached truncated stack-traces 1: locator_

2:

waitfor

olivier-martin-sf commented 1 year ago

Hi @tanujvishno, thank you for taking the time to open this issue.

Hey everyone, we're working with utam-webdriver-jasmine flavors along with TS

May you clarify what's utam-webdriver-jasmine? We don't have such package published on npm. Is it a package that you developed internally or is it terminology meaning that you are using WebdriverIO with Jasmine?

in-short we got a complete stack trace in case of assertion failure like which spec line and file, but other failures won't populate complete stack traces & the error source

Would you be able to set up a minimal repository where the unwanted behavior can be reproduced? That would be helpful to see a test that exhibits this behavior (it can be against any Web page and doesn't have to be against a Salesforce test environment).

tanujvishnoi commented 1 year ago

sorry for the confusion , Yes, by utam-webdriver-jasmine I meant webdriverIO with jasmine , I'll share a repo url soon reproducing this issue

tanujvishnoi commented 1 year ago

@olivier-martin-sf https://github.com/tanujvishnoi/utam_demo , here is the demo code to produce truncated stack trace

olivier-martin-sf commented 1 year ago

Thank you so much for providing the minimal repository, we are going to look into this as soon as we have the capacity to do so (probably next week)

olivier-martin-sf commented 1 year ago

Hi @tanujvishnoi, I look into this issue and I would like to clarify the problem statement and expectations.

Digging deeper, if I understood the problem statement correctly, it seems that the issue appears in the context of a waitFor predicate.

Getting non-existing element

From my observations, if I am trying to get a non-existing element, the stack trace provides the Jasmine context (with the position of the line that throws the error in the spec file). To reproduce try to define a public element with a CSS selector that doesn't match any element and try to invoke the function that gets the element from the test. For example:

{
  // other non-relevant properties
  "elements": [
    {
      "name": "userNameInput",
      "type": ["editable", "clickable"],
      "public": true,
      "selector": {
        "css": ".dsadsads"
      }
    }
  ],
  // other non-relevant properties
}

And try to invoke this from the test:

await demoPage.getUserNameInput();

This should throw an error with the following stack trace:

[0-0] Error: Can't find elements with locator 'dsadsads' inside its scope element.
[0-0]     at ElementWdioAdapter.findElements (/Users/JaneDoe/utam_demo/node_modules/wdio-utam-service/build/adapters/element-adapter.js:95:19)
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[0-0]     at async ElementWdioAdapter.findElement (/Users/JaneDoe/utam_demo/node_modules/wdio-utam-service/build/adapters/element-adapter.js:88:17)
[0-0]     at async Demo2.getUserNameInput (/Users/JaneDoe/utam_demo/pageObjects/demo2.js:44:23)
[0-0]     at async UserContext.f1 (/Users/JaneDoe/utam_demo/tests/spec/test2.spec.ts:9:7)

As you can see the last line of the stack trace is the Jasmine UserContext.

Note: f1 is the name of the function that wraps the test, I avoid arrow functions in tests to have more explicit context in the stack trace.

Getting non-existing element inside a predicate

However if I am trying to wait for an element that doesn't exist in a waitFor predicate such as:

// other non-relevant properties
"methods:"  [
   {
      "name": "waitForNonExistentElement",
      "compose": [
        {
          "apply": "waitFor",
          "args": [
            {
              "type": "function",
              "predicate": [
                {
                  "element": "root",
                  "apply": "containsElement",
                  "args": [
                    {
                      "type": "locator",
                      "value": {
                        "css": ".fdskjfdkslj"
                      }
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
]
// other non-relevant properties

And I am trying to invoke this method from my test:

await demoPage.waitForNonExistentElement();

This is the error that I am having:

[0-0] Error: Timeout while waiting for condition after 4639ms.
[0-0]     at handlePollTimeout (/Users/JaneDoe/utam_demo/node_modules/@utam/core/build/helper-methods.js:53:28)
[0-0]     at /Users/JaneDoe/utam_demo/node_modules/@utam/core/build/helper-methods.js:62:25
[0-0]     at processTicksAndRejections (node:internal/process/task_queues:96:5)

There we don't have the information about the test context (we don't know which line in our spec file generated the error).

tanujvishnoi commented 1 year ago

Hi @olivier-martin-sf

Thanks for sharing the examples . Yes the issue eventually reproducible for predicate . If a condition get failed inside a predicate , we did not get the full error stack Apart from the above 2 conditions, i added one more where we expect a hidden element to get visible . https://github.com/tanujvishnoi/utam_demo

`1) reproduce truncate error stack Case 1: Getting no existing element directly using getter Error: Can't find elements with locator 'dsadsads' inside its scope element. Error: Can't find elements with locator 'dsadsads' inside its scope element. at ElementWdioAdapter.findElements (\utam_demo\node_modules\wdio-utam-service\build\adapters\element-adapter.js:95:19) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async ElementWdioAdapter.findElement (\utam_demo\node_modules\wdio-utam-service\build\adapters\element-adapter.js:88:17) at async Demo.getUserNameInput (\utam_demo\pageObjects\demo.js:50:23) at async UserContext. (\utam_demo\tests\spec\test.spec.ts:12:21)

2) reproduce truncate error stack Case 2: Getting no existing element inside a method Error: Can't find elements with locator 'dsadsads' inside its scope element. Error: Can't find elements with locator 'dsadsads' inside its scope element. at ElementWdioAdapter.findElements (\utam_demo\node_modules\wdio-utam-service\build\adapters\element-adapter.js:95:19) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async ElementWdioAdapter.findElement (\utam_demo\node_modules\wdio-utam-service\build\adapters\element-adapter.js:88:17) at async Demo.getUserNameInput (\utam_demo\pageObjects\demo.js:50:23) at async Demo.clickUserName (\utam_demo\pageObjects\demo.js:85:29) at async UserContext. (\utam_demo\tests\spec\test.spec.ts:23:9)

3) reproduce truncate error stack Case 3: Getting no existing inside a predicate Error: Timeout while waiting for condition after 10484ms. Error: Timeout while waiting for condition after 10484ms. at handlePollTimeout (\utam_demo\node_modules\@utam\core\build\helper-methods.js:53:28) at \utam_demo\node_modules\@utam\core\build\helper-methods.js:62:25 at processTicksAndRejections (node:internal/process/task_queues:96:5)

4) reproduce truncate error stack Case 4: Wait for visibility of hidden search input Error: Timeout while waiting for condition after 10265ms. Error: Timeout while waiting for condition after 10265ms. at handlePollTimeout (\utam_demo\node_modules\@utam\core\build\helper-methods.js:53:28) at \utam_demo\node_modules\@utam\core\build\helper-methods.js:62:25 at runMicrotasks () at processTicksAndRejections (node:internal/process/task_queues:96:5)

5) reproduce truncate error stack Case 5: Get hidden element ,then wait for visibility Error: Timeout while waiting for condition after 10212ms. Error: Timeout while waiting for condition after 10212ms. at handlePollTimeout (\utam_demo\node_modules\@utam\core\build\helper-methods.js:53:28) at \utam_demo\node_modules\@utam\core\build\helper-methods.js:62:25`

olivier-martin-sf commented 1 year ago

Thanks, I will pull your changes.

Judging by your stack traces example, 1 & 2 provide the full stack trace but 3, 4 & 5 don't and all of them are related to timeout errors so can we narrow the problem statement to: "we don't have the full stack trace when there's a wait condition that times out?"

Do you agree with that? If yes, I will start investigating why we shallow the context in waitFor

tanujvishnoi commented 1 year ago

Yes absolutely agree.

olivier-martin-sf commented 1 year ago

After some investigation, I can confirm that I can reproduce the issue. It happens when the waitFor function is being invoked and the condition doesn't return a truthy value before the timeout.

The problem is due to the execution of an async function that is triggered by a timer as described here: https://github.com/sindresorhus/got/blob/main/documentation/async-stack-traces.md. I created an item in our backlog so that we can fix the issue in a future release.

Thank you for reporting this

tanujvishnoi commented 1 year ago

Hi @olivier-martin-sf just taking a follow up, please let us know once we have a fix for this issue

irinash13 commented 1 year ago

Hello @tanujvishnoi , what company do you represent?

KierenLWoods commented 1 year ago

@olivier-martin-sf Just wondered if there are any updates on this one and any plans as to when it may be fixed?

olivier-martin-sf commented 1 year ago

Hi @tanujvishnoi & @KierenLWoods, this issue is being tracked in our backlog but hasn't been prioritised yet.