Ability to dynamically create tests while inside of a test - cannot create tests while tests are running

[stevenlafl]

I am loading these based on some data, but to get it I need to run cy.task to grab that data. For now let's assume it's a MySQL query via a plugin. I cannot use cy.task outside of a test, so I am trying to add tests from within a test.

Another use case is that I am running a test, but I see a button. I have another set of tests that only run based on the existence of that button. I still want them in another separate context, so a separate dropdown.

Also, I cannot use it inside of a cy.task callback e.g. inside of myFunction like with cy.task('readCSV', 'test.csv').then(myFunction);

Current behavior:

Parent task results waits forever. Perhaps related (https://github.com/webdriverio/webdriverio/issues/4963)

Desired behavior:

I would like to see "Another Test" in a separate row as per the code below. FYI: Doing this will work in puppeteer correctly.

Test code to reproduce

context('Set up',  () => {
  it('add tests',  () => {

    context('Another test', () => {

      it('this title', () => {
        expect(true).to.equal(true)
      })
    })

  });
})

Versions

Debian 10, Cypress 4.8.0

[stevenlafl]

Actually, I set up a setTimeout recursive function and ran it outside of any tests. It appears that while tests are running, you can't add more.

Consider:

let results = [];

describe('Task 1',  () => {
  it('add tests',  () => {
    results = ['test'];
  });
})

describe('Task 2',  () => {
  it('add tests',  () => {

  })
})

let found = false;
const AddTestsAfterStart = (title, ms) => {
  const delay = 50
  if (results.length > 0) {

    describe('Dyanamic 1',  () => {
      it('test tests',  () => {
        expect(true).to.equal(true)
      });
    })

    describe('Dynamic 2',  () => {
      it('test tests',  () => {
        expect(true).to.equal(true)
      });
    })
  }
  else {
    setTimeout(() => {
      AddTestsAfterStart(title, ms - delay)
    }, 50)
  }
}

AddTestsAfterStart('test', 500);

function setTest(res) {
  results = res;
}

I get "Task 1" and "Task 2"

Now, if I change the line at the top:

let results = ['test'];

I get "Task 1" and "Task 2" and "Dynamic 1" and "Dynamic 2".

So, I can only conclude that you can only add tests before any tests run. This is a problem for me.

[stevenlafl]

https://github.com/cypress-io/cypress/issues/5418 works for reading from a file.

However, I am thinking that support for this, as Puppeteer has it, would be great for solving the potential case above: "I have another set of tests that only run based on the existence of that button. I still want them in another separate context, so a separate dropdown."

[josephzidell]

Cypress needs all tests registered to run them. Can you wrap the expects in the if conditions?

Something like:

    describe('Dyanamic 1',  () => {
      it('test tests',  () => {
        if (results.length > 0) {
          expect(true).to.equal(true)
        }
      });
    })

[stevenlafl]

Cypress needs all tests registered to run them. Can you wrap the expects in the if conditions?

Something like:

    describe('Dyanamic 1',  () => {
      it('test tests',  () => {
        if (results.length > 0) {
          expect(true).to.equal(true)
        }
      });
    })

I can and I do, but that isn't quite what I am looking for.

The issue is that I want the flexibility to add these tests at runtime. For example, I would like a test to

1. Grab site URLs that exist in a CSV
2. For each URL, run a set of browser tests them
3. For each test, add more tests depending on the result.

Depending on what is on a certain page, I may need to run additional tests, since the structure could be entirely arbitrary or user defined. Even structures such as content types and their fields are not known to me ahead of time.

The idea is achieve a write-once set of tests, and transport them for use across multiple site installations with a wide variety of scenarios as to their configuration.

These tests are entirely decoupled from the site installation and need to be run remotely. Multiple people are changing the structures all at the same time, so when these run, I need to be able to report back to them continuously.

[intelcentre]

Mocha has a command line option called "--delay". This would be great to implement in cypress for those of us who have async operations they need to perform which inform the list of actual tests.

From their docs:

DELAYED ROOT SUITE

https://mochajs.org/#asynchronous-code If you need to perform asynchronous operations before any of your suites are run, you may delay the root suite. Run mocha with the --delay flag. This will attach a special callback function, run(), to the global context:

setTimeout(function () {
  // do some setup

  describe('my suite', function () {
    // ...
  });

  run();
}, 5000);

[jennifer-shehane]

Due to the lack of interest in this feature over the years, we’ll close this as not planned currently.