No error feedback when using Cucumber and webdriverio

samhatoum commented 6 years ago

Issue by maxmckenzie Saturday Dec 31, 2016 at 15:24 GMT Originally opened as https://github.com/xolvio/chimp/issues/546

Chimp seems to abstract away the feedback from the testing framworks.

  @watch
  Scenario: As a user I want to add an event to the diary
    Signing out...
  - Given I open the url "/diary"

Failures:

1) Scenario: As a user I want to add an event to the diary - features/diary.feature:6
   Step: Before
   Step Definition: features/steps/global.js:10
   Message:
     TypeError: Cannot read property 'trim' of undefined
         at World.<anonymous> (features/steps/global.js:14:8)

I've used webdriver and cucumber before on their own and this would have returned something along the lines of could not find selector '.logout_link' The same goes for the ruby capybara port.

Surely we should pass these errors forward. It makes it a nightmare to work with.

or am I missing something here

here is the args I pass in.

"test": "chimp --chai --compiler=js:babel-register --jsonOutput=cucumber_output.json",

Expected behaviour

When I add a browser.click('selector') to my test steps And I run the test on a page where this selector does not exist Then I expect to be told in the command line that the element does not exist

Actual behaviour

When I add a browser.click('selector') to my test steps And I run the test on a page where this selector does not exist Then i get the error TypeError: Cannot read property 'trim' of undefined returned

Some example code for context of dependencies and run scripts

https://gist.github.com/maxmckenzie/475383187e989394c9d9e66a5c3a6128

Version & tools:

Chimp command line used: The exact command you used chimp --chai --compiler=js:babel-register --jsonOutput=cucumber_output.json
Chimp config file: Please provide a Gist if you use a config file
Chimp: type chimp -v 0.45.1
Node.js: type node -v v7.2.1
Java: type java -version java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
Operation system: type uname -v Darwin Kernel Version 16.3.0: Thu Nov 17 20:23:58 PST 2016; root:xnu-3789.31.2~1/RELEASE_X86_64

Console / Log Output

Join our Slack xolv.io/community #chimp channel, where you can find help and help others.

samhatoum commented 6 years ago

Comment by markelog Thursday Jan 05, 2017 at 14:45 GMT

It makes it a nightmare to work with.

It really is :(. Would be cool if this issue would be prioritized

samhatoum commented 6 years ago

Comment by samhatoum Thursday Jan 05, 2017 at 14:50 GMT

The issue above was solved. It was due to confusion with using async commands.

What's your specific case @markelog ?

samhatoum commented 6 years ago

Comment by markelog Thursday Jan 05, 2017 at 14:55 GMT

If it was resolved why this is open?

I'm constantly seeing those: TypeError: Cannot read property 'trim' of undefined returned when using click method on the element

samhatoum commented 6 years ago

Comment by markelog Thursday Jan 05, 2017 at 14:56 GMT

I probably can come up with some test-case, although it looks a bit time consuming

samhatoum commented 6 years ago

Comment by samhatoum Friday Jan 06, 2017 at 17:19 GMT

@maxmckenzie can you please post your answer of how we was resolved on slack for others to benefit? Much appreciated!

@markelog we would love to help you, but we need more info from you. Please open a new issue and see the template we setup that will help us help you. 👍

samhatoum commented 6 years ago

Comment by maxmckenzie Friday Jan 06, 2017 at 17:35 GMT

I'm actually still struggling with this issue of it just returning trim. I'm currently using the syncByDefault: false flag.

My test case code

this.Before(function () {
    console.log("   Signing out...");
    browser
      .url(url + '/')
      .click('.header__link .fa-key')
    const username = 'user'
    const password = 'password'
    browser
      .url(url + '/auth')
      .setValue('input[name="identifier"]', username)
      .setValue('input[name="password"]', password)
      .submitForm('form')
  })

has been refactored into

this.Before(function() {
    browser
      .url(url + '/auth/logout')
      .url(url + '/auth')
      .setValue('input[name="identifier"]', 'username')
      .setValue('input[name="password"]', 'password')
      .submitForm('form')
      .waitForExist('.dashboard', 10000)
  })

It works, but I still have the return trim error rather than "selector cannot be found"

@samhatoum is there a further more verbose feedback we can get from cucumber without having to go into the debugger. What usually gets returned when running cucumber.js outside of chimp?

@markelog time-consuming it may be, but without it we're just stabbing in the dark. it could be an issue on the way we have it set up. It may not be. If you open a new issue you'll see the issue template to fill out.

samhatoum commented 6 years ago

Comment by markelog Friday Jan 06, 2017 at 18:11 GMT

time-consuming it may be

I meant i'm not sure if I would find the time for it, so no promises, I wouldn't ignore such signals from the users anyhow

samhatoum commented 6 years ago

Comment by samhatoum Friday Jan 06, 2017 at 18:46 GMT

@maxmckenzie Sounds like it's promises related - that is we're gobbling up the errors that come up when chained in promises. I've reopened. We hardly see anyone using async!

@markelog I'm responding to your signal by asking for more information. Like, I don't know if you're using sync or async and I don't see any logs that can tell me anything about your case. If I can't understand the problem then I can't solve it.

So, in order to help you, I need your help in understanding and opening a new issue with the right info will allow me to understand the problem which will lead to me helping you.

samhatoum commented 6 years ago

Comment by markelog Friday Jan 06, 2017 at 19:05 GMT

If I can't understand the problem then I can't solve it!

No need for exclamation marks :), I get it, no worries, my issues is the same error as in this ticket, I was just pointing out that this is not an isolated case...

samhatoum commented 6 years ago

Comment by samhatoum Friday Jan 06, 2017 at 19:15 GMT

I've removed the exclamation mark! ;)

samhatoum commented 6 years ago

Comment by maxmckenzie Friday Jan 06, 2017 at 21:27 GMT

@samhatoum one thing i found confusing was, to turn off async you use syncByDefault = false

asyncByDefault = false would be a more straight forward flag no?

samhatoum commented 6 years ago

Comment by derwok Tuesday Feb 28, 2017 at 21:34 GMT

We were biten by this topic, also! We use chimp (with webdriver.io). When running the tests headless everything is fine. Wenn running the tests whitch chrome driver, then non-existing selectors lead to a very confusing error message. ;-( and debugging is very hard...

I created a minimal repo that shows the error: https://github.com/derwok/chimptest (The comprehensive README.md gives version numbers of all tools and steps to reproduce)

When running the tests with node 7.x and we hit this line: browser.click("#DOES_NOT_EXIST"); then the test runner crashes with:

     TypeError: Cannot read property 'trim' of undefined
      at sanitizeErrorMessage (/usr/local/lib/node_modules/chimp/node_modules/wdio-sync/build/index.js:190:34)
      at Object.<anonymous> (/usr/local/lib/node_modules/chimp/node_modules/wdio-sync/build/index.js:467:23)
      at Context.<anonymous> (tests/ChimpTest.js:13:17)
      at /usr/local/lib/node_modules/chimp/dist/lib/utils/fiberize.js:25:14

FunFact: node version matters!

It seems, the problem is strongly related to the used node version. Because when we switch to node 4.x (latest) or node 6.x (latest) we get the expected error:

 Error: An element could not be located on the page using the given search parameters.

samhatoum commented 6 years ago

Comment by lucetius Wednesday Mar 15, 2017 at 10:38 GMT

It seems like problem is related to wdio-sync not passing error.stack when using nodejs version 7.x For now we recommend using LTS version of nodejs.

samhatoum commented 6 years ago

Comment by pricetula Thursday Mar 16, 2017 at 07:21 GMT

[chimp][helper] setupBrowserAndDDP had error TypeError: Cannot read property 'trim' of undefined at sanitizeErrorMessage (/home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/node_modules/wdio-sync/build/index.js:87:34) at Object. (/home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/node_modules/wdio-sync/build/index.js:337:53) at Object. (/home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/dist/lib/session-manager.js:145:21) at initBrowser (/home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/dist/lib/chimp-helper.js:174:17) at Object.setupBrowserAndDDP (/home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/dist/lib/chimp-helper.js:248:7) at /home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/dist/lib/cucumberjs/hooks.js:21:24 at /home/.nvm/versions/node/v7.2.0/lib/node_modules/chimp/node_modules/meteor-promise/fiber_pool.js:32:39

So how can we make chimp run on previous versions of node??

samhatoum commented 6 years ago

Comment by pricetula Thursday Mar 16, 2017 at 08:16 GMT

I was running this command "chimp --ddp=http://localhost:3000 --watch --path=tests --browser=firefox" but changed this to "chimp --ddp=http://localhost:3000 --watch --path=tests --browser=phantomjs" And it now shows scenarios and steps.....what is difference between firefox and phantomjs i am new in e2e tests

samhatoum commented 6 years ago

Comment by maxmckenzie Thursday Mar 16, 2017 at 13:47 GMT

@pricetula what version of node are you using? you can type node -v into your terminal to find out

samhatoum commented 6 years ago

Comment by pricetula Thursday Mar 16, 2017 at 13:50 GMT

node is v7.2.0

samhatoum commented 6 years ago

Comment by maxmckenzie Thursday Mar 16, 2017 at 13:56 GMT

Yep ok, so as derwok said above. It seems to be an issue with node versions.

only even node version numbers are LTS (long term support) so switch to using node v6x latest

I have no idea why only even numbers are supported but i suppose its a system :D

samhatoum commented 6 years ago

Comment by samhatoum Friday Mar 17, 2017 at 11:32 GMT

I'm seeing this issue also now @lucetius and @twilczek

Ping me for a demo

samhatoum commented 6 years ago

Comment by RyanGannon Thursday Mar 23, 2017 at 14:31 GMT

Is it possible this is related to https://github.com/webdriverio/webdriverio/issues/1885 ? Needless to say I've been looking for the same solution.

Edit: I've gotten this working using Chimp v0.45.1 + Node v6.10.1

samhatoum commented 6 years ago

Comment by maxmckenzie Wednesday Mar 29, 2017 at 13:48 GMT

@RyanGannon good man, I can confirm that using those version pairs...

pairs? can you say pairs. Is that a thing?

anyway chimp v0.45.1 and node v6.10.1 works as expected.

samhatoum commented 6 years ago

Comment by maxmckenzie Wednesday Mar 29, 2017 at 13:50 GMT

@samhatoum should we add something to the chimp docs to state this?

samhatoum commented 6 years ago

Comment by jbblanchet Tuesday Sep 05, 2017 at 13:50 GMT

Sorry to bump this issue, but my understanding was that this was not a priority since it didn't happen in a LTS. Since 8.x LTS is maybe a month away, and this issue is still happening on 8.4. I was wondering if there was any progress on a fix.

TheBrainFamily / chimpy