Make grepFilterSpecs work when specPattern has been set in the config

[jablesyeah]

The grepFilterSpecs only appears to work correctly if the specPattern value is the default value. If the value is changed, by setting it within the config, the grepFilterSpecs no longer works.

example of config:

const { defineConfig } = require('cypress');

module.exports = defineConfig({
    e2e: {
        setupNodeEvents(on, config) {
            require('cypress-grep/src/plugin')(config);
            return config;
        },
        baseUrl: 'http://localhost:802/',
        specPattern: 'tests/cypress/e2e/**',
        supportFile: 'tests/cypress/support/e2e.js',
        env: {
            grepFilterSpecs: true,
            grepOmitFiltered: true,
        },
    },
});

With specPattern in config: Specs: 54 found

Without specPattern in config: Specs: 17 found

[joshuajtward]

@jablesyeah have you tried putting specPattern above setupNodeEvents (blind guess that defineConfig runs through the config sequentially)? And if that fails, what output do you get when running with DEBUG=cypress-grep in the CLI?

[jablesyeah]

Ordering makes no difference.

Can't get any logs to show passing the debug: DEBUG=cypress-grep yarn cypress run --config-file tests/cypress.config.js --env grepTags=@core

I can see that debug logs are coming through for cypress using this though: DEBUG=cypress:* yarn cypress run --config-file tests/cypress.config.js --env grepTags=@core

Couldn't see any reference to cypress grep in those debug logs though.

[joshuajtward]

One last thing before I try to recreate it locally: does it work if you move the env config outside e2e?

[jablesyeah]

Still persists, I'm afraid. The only thing I can do to make it work is to remove the specPattern from the config. Then cypress will run with the grepTags correctly but will cause an error if run without the grepTags, as cypress cannot find the test files using the default specPattern value.

[gitlax]

I had to set my grepIntegrationFolder: "../../".

From what I could tell, the place where it was looking for my specs was in the config directory. Without changing the code for the plugin (which would work for me locally but not on my CD server) that was the only way I could make the grepFilterSpecs work.

[jablesyeah]

I had to set my grepIntegrationFolder: "../../".

Not sure what you mean here @gitlax . I have tried adding the grepIntegrationFolder value into my config (i didn't even realise this value existed), but it made no difference for me. not in the env, e2e or the base module.exports.

[gitlax]

I had to set my grepIntegrationFolder: "../../".

Not sure what you mean here @gitlax . I have tried adding the grepIntegrationFolder value into my config (i didn't even realise this value existed), but it made no difference for me. not in the env, e2e or the base module.exports.

It is very possible that I have something configured incorrectly. What was happening to me was every time I would run my tests it would tell me that all specs had been filtered out. Doing a little debugging I found that the folder it was looking for the specs was the config folder rather than the e2e folder. The plugin aggregates all the files that match the specPattern (default is cypress/e2e/*/.cy.{js,jsx,ts,tsx}). Since it was starting in the config folder, it was looking for tests in "cypress/config/cypress/e2e/*/.cy.{js,jsx,ts,tsx}". Which did not exist. So the set was empty, and all specs were "filtered out". To get it to work, I told the grepIntegrationFolder to start two directories higher. This allowed it to find my tests. Not sure what I may have that is different that is causing it to think its CWD is the config folder.

[jablesyeah]

@gitlax This sounds like a separate issue to me.

[jablesyeah]

@joshuajtward Have you managed to recreate this?

[GilbertoGNG]

I'm having the same issue. grepFilterSpecs: true is not working properly with my installation of Cypress 10.2.0.

Is there something we can do to help you guys out?

[bahmutov]

To speed up solving this issue a small repo showing the problem is 200x useful

[joshuajtward]

@joshuajtward Have you managed to recreate this?

No, I have not been able to recreate it, the only time I could see anything kinda similar was if I commented out the registration in the e2e.js support file. I'm wondering if you could be having install issues, or you have another plugin clashing with cypress-grep as you should be seeing debug logs from the command you posted above (screenshot here for reference of what they usually look like):

As @bahmutov says, it's significantly quicker for others to help get to the root of these issues if you can recreate it in a minimum-size repo, it may also help you iterate more quickly when debugging. You can fork the repo I used to try and recreate your issue, but you'll probably need to tweak it to simulate your setup better: https://github.com/joshuajtward/cypress-grep-testing

[s4ms00n9]

@bahmutov I was able to reproduce the issue - the only thing that seems to make it buggy is having a config file in other directory than the default one (root folder). @jablesyeah I can see in one of your first comments that this could be the issue, right? You have the config file inside tests/ directory.

[jablesyeah]

@s4ms00n9 @bahmutov @joshuajtward That's right. It appears to be related to the location of the config file.

[jablesyeah]

@joshuajtward Now we have found what this issue is, do you think it's something that can be fixed?

[joshuajtward]

Hi @jablesyeah! I can investigate this, but from the surface level it's a bit of an odd one - the way this part of the plugin works is that the resolved Cypress config is passed to the plugin (via require('cypress-grep/src/plugin')(config)) so I wouldn't expect the location of the config file to make a difference

Asking out of ignorance - is there a specific reason to have the config file in a different location?

I'll see if I can carve out some time this weekend to have a dig around...

[jablesyeah]

@joshuajtward Awesome. I appreciate you looking into this.

I believe its something to do with our docker images etc but is an implementation made by the devs, so I'm not 100% sure.

[s4ms00n9]

Hi @jablesyeah! I can investigate this, but from the surface level it's a bit of an odd one - the way this part of the plugin works is that the resolved Cypress config is passed to the plugin (via require('cypress-grep/src/plugin')(config)) so I wouldn't expect the location of the config file to make a difference

Asking out of ignorance - is there a specific reason to have the config file in a different location?

I'll see if I can carve out some time this weekend to have a dig around...

My case is that I have several config files for different environments and I store them in a separate folder (which makes the location not default). And maybe as a hint - it is not mentioned in the official docs, but I think adding grepIntegrationFolder: "../../" (pointing to a root folder from where the config file is) into env may be a workaround for this issue.

[joshuajtward]

Yeah good point @s4ms00n9, grepIntegrationFolder is a bit of an artefact from when I was doing the upgrade to work with 10.0.0, I initially wrote it to be backward compatible with 9.0.0, but I was persuaded to make it a breaking change to gently push people to upgrade. I think that's why I forgot to add any info to the README about it 🤦‍♂️

For this issue going forward, I think grepIntegrationFolder should probably be renamed to something like relativeConfigFileToIntegrationFolderPath (or use path to resolve two paths for us), and I should add something to the docs for it. @jablesyeah does setting it to be the relative path for your setup work? (not as a solution, but to validate that we're on the right track)

In addition, I've just seen this: cypress-io/cypress-grep#153, so we should probably move this issue to the main Cypress repo, as cypress-grep excitingly now lives here: https://github.com/cypress-io/cypress/tree/develop/npm/grep