@cypress/grep - grepFilterSpecs and grepOmitFiltered have no effect

[darioprinc]

Root cause:

When the @cypress/grep plugin appears to use the location of the cypress config file as the path for the integrationFolder. If the config file is not at the root of the project, then it does not find the specs correctly based on this path and the spec pattern.

---

Original issue

Current behavior

When using grep flag, tests with a certain name are correctly ran. However using grepFilterSpecs and grepOmitFiltered has no effect.

Repro:

1. register plugin in both cypress/cypress.config.js and cypress/support/e2e.js according to the guide
2. set env in cypress.config.js:

   env: {
   grepFilterSpecs: true,
   grepOmitFiltered: true,
   grep: 'The Login Page'
   }

Note: There is only one test containing 'The Login Page' in it's name:

3. run from the terminal:

   set DEBUG=@cypress/grep && npx cypress run --config-file cypress/cypress.config.js

   ---

Result: 'The Login Page' test is being correctly ran, other tests are being incorrectly "skipped" - they pass with 0 tests:

Desired behavior

Expected: Only the test with the provided name is being ran. The rest is completely omited when grepFilterSpecs and grepOmitFiltered are enabled and no time is being wasted on going through "skipped" tests.

Test code to reproduce

Please see above.

Cypress Version

12.11.0

Node version

v16.14.0

Operating System

Microsoft Windows 10 Pro, 10.0.19044

Debug Logs

@cypress/grep @cypress/grep plugin version 3.1.5 +0ms
@cypress/grep: tests with "The Login Page" in their names
@cypress/grep: will omit filtered tests
  @cypress/grep Cypress config env object: { FAIL_FAST_ENABLED: false, grepFilterSpecs: true, grepOmitFiltered: true, grep: 'The Login Page', apiUrl: MASKED, baseUrl: MASKED, emberBaseUrl: MASKED, numTestsKeptInMemory: 1, __cypress_cucumber_preprocessor_dont_use_this_suite: { isEventHandlersAttached: true } } +0ms
  @cypress/grep specPattern [
  'cypress/e2e/**/*.cy.{js,jsx,ts,tsx}',
  'cypress/e2e/cucumber/**/*.feature'
] +1ms
  @cypress/grep excludeSpecPattern cypress/e2e/cucumber/**/*.{js,ts} +0ms
  @cypress/grep integrationFolder C:\web\MASKED\monorepo\cypress +0ms
  @cypress/grep found 0 spec files +5ms
  @cypress/grep [] +0ms
@cypress/grep: filtering specs using "The Login Page" in the title
  @cypress/grep parsed grep { title: [ { title: 'The Login Page', invert: false } ], tags: [] } +1ms
  @cypress/grep found grep "The Login Page" in 0 specs +0ms
  @cypress/grep [] +0ms
grep and/or grepTags has eliminated all specs

[warrensplayer]

@darioprinc Can you setup a minimal reproduction of this issue? When I follow the steps you provided, I can get my tests to filter as expected. I do see a difference in the debug logs for the @cypress/grep plugin. I can see it iterate through my spec files:

@cypress/grep @cypress/grep plugin version 3.1.5 +0ms
  @cypress/grep Cypress config env object: { grep: 'The Login Page', grepFilterSpecs: true, grepOmitFiltered: true } +0ms
@cypress/grep: tests with "The Login Page" in their names
@cypress/grep: will omit filtered tests
  @cypress/grep specPattern cypress/e2e/**/*.cy.{js,jsx,ts,tsx} +1ms
  @cypress/grep excludeSpecPattern *.hot-update.js +0ms
  @cypress/grep integrationFolder /Users/stokes/Development/firewatch/26642 +0ms
  @cypress/grep found 2 spec files +5ms
  @cypress/grep [ '/Users/stokes/Development/firewatch/26642/cypress/e2e/spec.cy.js', '/Users/stokes/Development/firewatch/26642/cypress/e2e/spec2.cy.js' ] +0ms
@cypress/grep: filtering specs using "The Login Page" in the title
  @cypress/grep parsed grep { title: [ { title: 'The Login Page', invert: false } ], tags: [] } +0ms
  @cypress/grep spec file /Users/stokes/Development/firewatch/26642/cypress/e2e/spec.cy.js +6ms
  @cypress/grep suite and test names: [ 'The Login Page', 'passes 1' ] +0ms
  @cypress/grep spec file /Users/stokes/Development/firewatch/26642/cypress/e2e/spec2.cy.js +1ms
  @cypress/grep suite and test names: [ 'template spec', 'passes 2' ] +0ms
  @cypress/grep found grep "The Login Page" in 1 specs +1ms
  @cypress/grep [ '/Users/stokes/Development/firewatch/26642/cypress/e2e/spec.cy.js' ] +0m

A reproduction would help us find the issue much quicker. Thanks!

[warrensplayer]

Here is my reproduction. Can you reproduce your error with this?

26642.zip

[darioprinc]

hi @warrensplayer, I could not reproduce the error with your setup.

I created a reproducible example here:

cypress-sample-repo.zip

please run:

yarn
yarn cypress:run

[warrensplayer]

@darioprinc Thanks for the recreation. The difference between our two projects that was causing the issue is the location of the cypress config file. With my project, it was at the root of the project. When showing the debug messages from the grep plugin, you could see the different in this line:

with config under ./cypress

@cypress/grep integrationFolder /Users/stokes/Development/firewatch/26642/cypress-sample-repo-to-zip/cypress

with config under root

@cypress/grep integrationFolder /Users/stokes/Development/firewatch/26642/cypress-sample-repo-to-zip

The difference is the additional /cypress at the end of the path. We will have to investigate where Cypress is determining this integrationFolder relative to the location of the cypress config file.

As a workaround, you can move your cypress config file to the root of the project and the grep plugin should work as you expect.

[darioprinc]

@warrensplayer thank you. I know it is not really in scope of the opened issue, but after moving the cypress.config.js to the root I simply can't make the files to load. I am struggling with it for quite a few days now, maybe there is an obvious problem I am overlooking and that you've seen before?

First I started to get the import error:

import { addCucumberPreprocessorPlugin } from '@badeball/cypress-cucumber-preprocessor'; ^^^^^^ SyntaxError: Cannot use import statement outside a module

That would be relatively easily solved via adding "type": "module" in the root package.json (although it wasn't needed before or even present anywhere before, confusing).

This is one of my cypress.config.js imports:

import { getFileByTag } from '@support/grepTags/getTags';

And it results in the following error:

Error: Cannot find package '@support/grepTags' imported from C:\web\monorepo\cypress.config.js

@support would be an alias to the files in cypress/support folder to avoid those pesky ../../.. relative imports.

I have it defined in cypress/tsconfig.json:

{
  "compilerOptions": {
    "allowJs": true,
    "strict": true,
    "noEmit": true,
    "lib": ["es5", "es6", "es2019", "dom", "ES2021.String"],
    "types": ["cypress", "cypress-real-events"],
    "baseUrl": ".",
    "paths": {
      "@fixtures/*": ["fixtures/*"],
      "@support/*": ["support/*"]
    }
  },
  "include": ["./**/*", "../cypress.config.js"]
}

And even IDE recognizes the linking from cypress.config.js to @support files - I can click through one file to another, which only makes the error more confusing.

[nischaytv]

Did anyone find solution or any workaround for this?

[AlexandreRozier]

Thanks for this nice plugin, I'm also having this issue with

• @cypress/grep version: 4.0.1
• cypresss version 13.6.1

I have a cypress config file in cypress/cypress.local.config.ts

My test :

//cypress/e2e/test.cy.ts
it(`Does stuff`, { tags: '@fast' }, () => {...})

When using npx cypress run --config-file cypress/cypress.local.config.ts --env grepTags=@fast

Logs :

@cypress/grep @cypress/grep plugin version 4.0.1 +0ms
  @cypress/grep Cypress config env object: { grepTags: '@fast' ,grepOmitFiltered: true, grepFilterSpecs: true} +0ms
@cypress/grep: filtering using tag(s) "@fast"
@cypress/grep: will omit filtered tests
  @cypress/grep parsed grep tags [ [ { tag: '@fast', invert: false } ] ] +1ms
  @cypress/grep specPattern cypress/e2e/**/*.cy.{js,jsx,ts,tsx} +1ms
  @cypress/grep excludeSpecPattern *.hot-update.js +0ms
  @cypress/grep integrationFolder /home/rozierale/Work/Rte/Naza/alz/naza-front/cypress +0ms
  @cypress/grep found 0 spec files +5ms
  @cypress/grep [] +0ms
  @cypress/grep parsed grep tags { title: [], tags: [ [ [Object] ] ] } +0ms
  @cypress/grep found grep tags "@fast" in 0 specs +0ms
  @cypress/grep [] +0ms
grep and/or grepTags has eliminated all specs
grepTags: @fast
Will leave all specs to run to filter at run-time

---> This does not select test.cy.ts and filters the others, cypress-grep manually filters tests during test execution

If I move my cypress file to the root folder:

@cypress/grep @cypress/grep plugin version 4.0.1 +0ms
  @cypress/grep Cypress config env object: { grepTags: '@fast', grepOmitFiltered: true, grepFilterSpecs: true } +0ms
  @cypress/grep: filtering using tag(s) "@fast"
  @cypress/grep parsed grep tags [ [ { tag: '@fast', invert: false } ] ] +1ms
  @cypress/grep specPattern cypress/e2e/**/*.cy.{js,jsx,ts,tsx} +0ms
  @cypress/grep excludeSpecPattern *.hot-update.js +0ms
  @cypress/grep integrationFolder /home/rozierale/Work/Rte/Naza/alz/naza-front +0ms
@cypress/grep: will omit filtered tests
  @cypress/grep found 23 spec files +6ms
  ...
@cypress/grep spec file /home/rozierale/Work/Rte/Naza/alz/naza-front/cypress/e2e/test.cy.ts +1ms
  @cypress/grep test info: [ { type: 'test', pending: false, name: 'Does stuff' , tags: [ '@fast' ]} ] +0ms
  ...
  @cypress/grep found grep tags "@fast" in 1 specs +0ms
  @cypress/grep [ '/home/rozierale/Work/Rte/Naza/alz/naza-front/cypress/e2e/test.cy.ts' ]

---> This does select test.cy.ts and works as expected

Note: it is also mandatory to return configin setupNodeEvents(config, on) for grepOmitFiltered to work.

[asalgado87]

I'm seeing the same issue. This was running fine before we upgraded to typescript 5. I'm running my cypress projects in an nx monorepo. I have everything configured according to the documentation, but it still won't pre filter tests. This makes my test run take around 9 minutes instead of the 2 that it should.

running cypress 13.9.0 and cypress grep 4.0.1

[AnnemarieD]

Same here. It also worked fine until it didn’t anymore. We’re still trying to find out what the difference is between the settings before and after and why this is causing issues. This causes my pipelines to take around 10 minutes longer in total (5 minutes per environment).

[adoshi-branch]

Is there any update on this issue?

[AnnemarieD]

I got it working again by moving some config. I noticed that when running the command for the pipeline and explicitly stating grepTags, grepFilterSpecs and grepOmitFiltered it would work as expected. But when getting these options from our config it wouldn’t. Upon inspection of the settings when running cypress open I could also see that it wasn’t getting the expected options from the config.

We have a setup with a shared config for Cypress, and then config per (OTAP) environment that is based upon the shared config with some additional env options. The env options in the shared config seem to be overwritten by the env options in the specific config instead of combined. I moved the grep env options to the environment specific config and now it works again.