Setting per-test environment variables does not merge with per-suite envs & leaks to next test

[bahmutov]

Cypress v4.10.0

Reproduction https://github.com/cypress-io/cypress-example-recipes recipe examples/server-communication__env-variables spec file

The test sets per-suite environment variables and per-test environment variables.

context('Suite env variables', {
  env: {
    suiteApi: 'https://staging.dev',
    commonFlag: 'suite',
  },
}, () => {
  it('has all environment variables', () => {
    expect(Cypress.env('suiteApi')).to.equal('https://staging.dev')
  })

  // NOTE: does not work, seems test variables override
  // the suite variables but in a weird way (even after commenting out and
  // reloading the old variable is still there!)
  it.only('has test-specific env variables', {
    env: {
      testFlag: 42,
      commonFlag: 'test',
    },
  }, () => {
    expect(Cypress.env('testFlag'), 'test level variable').to.equal(42)
    expect(Cypress.env('commonFlag'), 'test overrides suite').to.equal('test')
    expect(Cypress.env('suiteApi'), 'suite level variable').to.equal('https://staging.dev')
  })
})

I expect the final Cypress.env object to have the variables from both the suite and the test itself

But it does not have suite variables

[bahmutov]

Second problem

I think it might be caused by the above and the logic we use to merge env variables. The variable from one test can "leak" into the other test

it('has its own variable 1', {
  env: {
    testOne: true,
  },
}, () => {
  expect(Cypress.env()).to.include({
    testOne: true,
  })
})

// NOTE: leaking variable from previous test
// https://github.com/cypress-io/cypress/issues/8005
it.skip('has its own variable 2', {
  env: {
    testTwo: true,
  },
}, () => {
  expect(Cypress.env(), 'has variable from this test').to.include({
    testTwo: true,
  })

  cy.log(Object.keys(Cypress.env()).join(', '))
  .then(() => {
    expect(Cypress.env(), 'does not have variable from first test').to.not.have.property('testOne')
  })
})

Here is the video how to get it to crash. First, run just the first test, then run both tests together and see variable from the first test in the second test

[ShamiliArj]

Can we expect this can be resolved or using this way is wrong

[emilyrohrbough]

@bahmutov I was able to confirm the issues / scenarios included on this issue with Cypress 9.2.1. This is still an issue.

[bahmutov]

Huzza, I mean ohhh

Sent from my iPhone

On Jan 12, 2022, at 10:34, Emily Rohrbough @.***> wrote:

 @bahmutov I was able to confirm the issues / scenarios included on this issue with Cypress 9.2.1. This is still an issue.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[mccataldo]

@bahmutov I'm running into a problem on v.9.3.1 where I can override an env var but the leak happens on retry. I'd like to help resolve this or find a workaround.

[bahmutov]

That’s unfortunate Mike, but I don’t have time to look into this

Sent from my iPhone

On Feb 8, 2022, at 16:04, mike cataldo @.***> wrote:

 @bahmutov I'm running into a problem on v.9.3.1 where I can override an env var but the leak happens on retry. I'd like to help resolve this or find a workaround.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[mccataldo]

@bahmutov It's OK. For now. I just reset env variables in support/index.ts:

Cypress.env('someVariableToOverrideAtSuiteLevel', 'itsDefaultValue')

[theKashey]

Oh, hopefully, that's just a bug. I was going to apply some terrible workarounds to play around this limitation. As I understand the problem - it's as simple as adding a missing use case for the First Problem at https://github.com/cypress-io/cypress/blob/1305cca9f19a1dd15851a3830209ba7597404777/packages/driver/cypress/integration/e2e/testConfigOverrides_spec.js

I can see a test covering Second problem added back in the original PR, and wondering why the second problem might exist.

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[theKashey]

Still a problem 🤷‍♂️

[tharari]

I think I found a workaround that I added to e2e.ts. It works just fine but I'm just not too sure if this is reliable. Can someone confirm whether this is a valid workaround?

Cypress.on("test:before:run", (test) => {
  const testLevelEnvironmentVariableOverrides =
    test._testConfig.testConfigList.find(
      (testConfig: { overrideLevel: string }) =>
        testConfig.overrideLevel === "test"
    ).overrides?.env;
  const suiteLevelEnvironmentVariableOverrides =
    test._testConfig.testConfigList.find(
      (testConfig: { overrideLevel: string }) =>
        testConfig.overrideLevel === "suite"
    ).overrides?.env;

    // use test level environment variabels if they exist, otherwise use suite level environment variables
  Cypress.env({
    ...(suiteLevelEnvironmentVariableOverrides || {}),
    ...(testLevelEnvironmentVariableOverrides || {}),
  });
});