search

cypress-io / cypress-grep

Filter tests using substring
137 stars 19 forks source link

grepTags not being obeyed on Windows? #93

Closed jdborneman-terminus closed 2 years ago

jdborneman-terminus commented 2 years ago

Running my tests (a very small/simple POC to prove out Cypress for our use cases) through a GitHub Action so that they run on MacOS: Chrome/FF and Windows: Chrome/FF/Edge.

MacOS tests run perfectly with my @SMOKE+@UI filter

Run yarn run cy:headless --browser chrome --env USERNAME="env.USERNAME",PASSWORD="env.PASSWORD",grepTags="@SMOKE+@UI" --config video=false,screenshotOnRunFailure=false
It looks like this is your first time using Cypress: 9.0.0

[STARTED] Task without title.
[SUCCESS] Task without title.

Opening Cypress...
[2605:1122/172029.222834:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[2605:1122/172029.222912:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[2605:1122/172029.222925:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[2605:1122/172029.222933:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
cypress-grep: filtering using tag "@SMOKE+@UI"
cypress-grep: will omit filtered tests

Run output:

(Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  0-terminus/terminus.ts                   00:08        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  1-google/google.ts                       00:03        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  2-failing/failing.ts                       2ms        -        -        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  3-api/reqbin.ts                            2ms        -        -        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  3-api/time.ts                              1ms        -        -        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:11        2        2        -        -        -

Windows tests ignore grepTags and run all tests. 😞 

Run yarn run cy:headless --browser chrome --env USERNAME="env.USERNAME",PASSWORD="env.PASSWORD",grepTags="@SMOKE+@UI" --config video=false,screenshotOnRunFailure=false
It looks like this is your first time using Cypress: 9.0.0

[STARTED] Task without title.
[SUCCESS] Task without title.

Opening Cypress...

[1168:1122/165618.283:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[1168:1122/165618.283:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[1168:1122/165618.283:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[1168:1122/165618.283:ERROR:node_bindings.cc(276)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
cypress-grep: will omit filtered tests

Note the missing cypress-grep: filtering using tag "@SMOKE+@UI" line for the Windows runs.

Run Output:

(Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  0-terminus/terminus.ts                   00:11        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  1-google/google.ts                       00:09        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  2-failing/failing.ts                     00:07        1        -        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  3-api/reqbin.ts                          426ms        3        3        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  3-api/time.ts                            358ms        2        2        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 5 failed (20%)                      00:29        9        8        1        -        -

It ran them all. 😞

I don't have a physical Windows machine to test this on, alas. I am running this with yarn as a Zero-Installs app (I think that's where the ERROR:node_bindings.cc is coming from) but they don't seem to impact running on MacOS.

Any pointers as to what I might be doing wrong? I wouldn't be surprised if it's something fiddly about envs passed in for Windows being problematic

bahmutov commented 2 years ago

I don't have a Windows machine to investigate, but try getting the debug logs https://github.com/cypress-io/cypress-grep#debugging to see what is going on. Also, try putting the grep tags first in the --env parameter?

jdborneman-terminus commented 2 years ago

I don't have a Windows machine to investigate, but try getting the debug logs https://github.com/cypress-io/cypress-grep#debugging to see what is going on. Also, try putting the grep tags first in the --env parameter?

Strange! Looks like moving grepTags to be first in the --env worked. Quite strange but I'll take it. I'll be closing this issue then. Thanks @bahmutov !

VolodymyrLykhachov commented 2 years ago

@jdborneman-terminus I think problem was with quotes. You could try to add quotes to whole --env parameter like: --env "USERNAME=env.USERNAME,PASSWORD=env.PASSWORD,grepTags=@SMOKE+@UI" or play with quotes