Component tests failing intermittently with uncaught Vite "failed to fetch dynamically imported module" error in CI

[synaptiko]

Current behavior

We started migrating our project from CRA to Vite and we have almost 300 component and 40 E2E Cypress tests in place. Unfortunately, after fixing all the other issues, we are still not able to stabilize our tests since there is always 1 or 2 failing randomly on "Failed to fetch dynamically imported module" errors.

We noticed that it's somehow related to the load of CI. Under some conditions more tests are failing like this, in other times it succeeds. But it's random. We checked our tests and we are pretty sure it's not caused by any logic we have.

We've checked some of the existing issues on Cypress & Vite, tried various workaround but no luck with any of them.

What we think is happening is that Cypress is not waiting for Vite to "boot up" properly and retries don't help with it, only when new spec is run, it works.

Note: it only happens with component tests. For E2E tests we had similar stability issues but we solved them by building production version and then just serving it with vite preview. This made integration tests faster and very stable. Previously they were also timeouting.

Note 2: we have a lot of components and lot of dependencies in our project, we also use MUI as our base. But with CRA we were able to have stable tests, it was just around two times slower. That's why we want to use Vite now.

Note 3: we are running "Component tests" in parallel, currently in 4 groups.

Desired behavior

No random problems with Cypress + Vite combo.

Test code to reproduce

Unfortunately can't provide this since our project is private. And I'm afraid that it's related to project's complexity and that's why we can't easily create another one for reproduction.

Cypress Version

v12.6.0

Node version

v16.18.1

Operating System

Docker Hub image: cypress/included for v12.6.0 version

Debug Logs

No response

Other

No response

[mschaefer-gresham]

We are having the exact same issue with component tests ( I was just coming here to open a new issue ) . We just migrated from webpack to vite. The error we get is:

The following error originated from your test code, not from Cypress. > Failed to fetch dynamically imported module: http://localhost:3000/__cypress/src/cypress/support/component.ts When Cypress detects uncaught errors originating from your test code it will automatically fail the current test. Cypress could not associate this error to any specific test. We dynamically generated a new test to display this failure.

We are getting this despite the fact that the file does exist. And this occurs randomly.

Vite: 4.1.1 Cypress: 12.5.1 Node: 19.3.0

One related issue: our cypress directory is on the same level as our src directory. When running locally, Cypress correctly expects component.ts to be where we have it in the cypress directory. But when we run the tests in docker, Cypress expects component.ts to be in a cypress directory under the src directory (see above error). Even if we use the 'supportFolder' config setting, Cypress still looks for it in the src directory (src is the vite root folder). So I just copied component.ts to the location cypress expects it to be (but still fails randomly despite this).

So locally this is not reproducable for us (so far). This only occurs in a docker container.

We are also running the tests in parallel.

[synaptiko]

Update: we found an ugly workaround by using https://docs.cypress.io/guides/guides/module-api#cypressrun and writing our own parallelization mechanism and runner which wraps Cypress. We are checking for errors and we will retry failed tests again.

It works but it requires some "balance" in the number of workers. We also made the first group run first, and only when it's successful, we run the other groups in parallel. This seems to help a lot. But there are still random failures from time to time but because we retry them a few times, it will eventually settle down.

It looks to be somehow related to "background" Vite's compilation & deps optimization because we observed that it usually gets stable once some of Vite's log messages appear.

[lmiller1990]

Is this only in Docker? We use Vite heavily internally - everything is really stable. I hope someone can reproduce it reliably.

I wonder if adding entries to https://vitejs.dev/config/dep-optimization-options.html#optimizedeps-entries helps?

[mschaefer-gresham]

For us it’s only in docker.

---

From: Lachlan Miller @.> Sent: Friday, February 24, 2023 4:11:38 AM To: cypress-io/cypress @.> Cc: Matthew Schaefer @.>; Comment @.> Subject: Re: [cypress-io/cypress] Latest Cypress + Vite causing unstable tests with "Failed to fetch dynamically imported module" errors (Issue #25913)

Is this only in Docker? We use Vite heavily internally - everything is really stable. I hope someone can reproduce it reliably.

I wonder if adding entries to https://vitejs.dev/config/dep-optimization-options.html#optimizedeps-entrieshttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fvitejs.dev%2Fconfig%2Fdep-optimization-options.html%23optimizedeps-entries&data=05%7C01%7Cmschaefer%40greshamtech.com%7C00633b16813e45580f2808db1614d876%7C4aed35c0c2db42c68c17efca15bfabfb%7C1%7C0%7C638128051020349418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=122w9yiXXe%2FXTHsrcXNC%2FQtuxpZZSZw3VYBCQYh05ao%3D&reserved=0 helps?

— Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcypress-io%2Fcypress%2Fissues%2F25913%23issuecomment-1442740369&data=05%7C01%7Cmschaefer%40greshamtech.com%7C00633b16813e45580f2808db1614d876%7C4aed35c0c2db42c68c17efca15bfabfb%7C1%7C0%7C638128051020349418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IjxbyfW3zqwlnheiv0cBiVQiLJA%2Ft%2BfLZk29kac3JZU%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAWR6LEEETUVEBY4J2GIRDNTWZARGVANCNFSM6AAAAAAVEDCD2Q&data=05%7C01%7Cmschaefer%40greshamtech.com%7C00633b16813e45580f2808db1614d876%7C4aed35c0c2db42c68c17efca15bfabfb%7C1%7C0%7C638128051020349418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Pl3T5QIWFTolfzMuSsUUwR9i4oNdBJIcl6Zv4ArHxRA%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>

[synaptiko]

Is this only in Docker?

Yes, it seems to be only happening on CI, which means Docker. And we tried optimizeDeps.entries but it didn't seem to really help.

It is somehow related to CI being under load. We observed a few more different errors happening. It just seems that loading of some resources is either failing or incomplete causing various strange issues (sometimes with loading the import, sometimes errors on Cypress level, sometimes in app, usually something about missing context). But all these are usually resolved after 1-3 retries, depending on the CI load.

[mschaefer-gresham]

We are experiencing more or less exactly the same thing. In particular the missing context error from time to time and failing to load component.ts.

[lmiller1990]

Maybe related: https://github.com/cypress-io/cypress/issues/18124#issuecomment-1445523806

The OP says "Latest Cypress + Vite" - is this suggesting this is a regression? If so, knowing which version introduced this regression would be very useful.

I haven't seen this in our internal suite, but it sounds like I may be able to reproduce by adding lots of large dependencies, like MUI? Or by increasing the load, eg Docker?

[mschaefer-gresham]

We use MUI. We have split approx 700 component tests into 8 groups of varying sizes which run in parallel in docker containters. The machine is sufficient for the load. Since writing the first post above we had 7 successful runs and then today we experienced the same failure across several of the groups. This time it was component.ts in one group, but also random test files in other groups. I tried the suggestion that @synaptiko gave to run one small group first before running the other tests in parallel. I have since had two successful test runs in a row. Hopefully this will stabalize the build more, but I have seen in the past that making a change can can improve things temporarily only to have a regression.

I have also tried running all the tests sequentially and get the same error btw.

[synaptiko]

@lmiller1990

The OP says "Latest Cypress + Vite" - is this suggesting this is a regression? If so, knowing which version introduced this regression would be very useful.

We are transitioning from CRA and much older Cypress, so I can really say if it's regression or not. We just started using versions from the last week.

it sounds like I may be able to reproduce by adding lots of large dependencies, like MUI? Or by increasing the load, eg Docker?

This correlates with our observations. We have MUI but also Syncfusion and a few other bigger dependencies. Our CI uses quite powerful machines but we are running a lot of things in parallel, the load changes over the day, which would explain why it happens randomly (but we observed it's like ~50% of cases).

[synaptiko]

For anyone interested, here's our "workaround" solution. We have a script called cypress-ct-runner.mjs (under bin/vite-migration:

#!/usr/bin/env node
import { Chalk } from 'chalk';
import cypress from 'cypress';
import fs from 'fs';
import path, { dirname } from 'path';
import { fileURLToPath } from 'url';

const chalk = new Chalk({ level: 1 });
const groupId = process.argv[2];

if (groupId === undefined) {
  console.error('Please provide a group id parameter');
  process.exit(1);
}

const __dirname = dirname(fileURLToPath(import.meta.url));
const groups = JSON.parse(fs.readFileSync(path.join(__dirname, '..', '..', 'component-test-groups.json')));
const group = groups[groupId - 1];
const maxRetries = 5;
let specs = group.join(',');
let shouldFail = false;
let shouldRetry = true;
let retries = 0;

while (shouldRetry) {
  if (retries > 0) {
    console.log(chalk.bgRed(`Retry number: ${retries}`));
    console.log(chalk.bgRed('Retrying failed tests:'));
    console.log(
      chalk.red(
        specs
          .split(',')
          .map((spec) => `- ${spec}`)
          .join('\n')
      )
    );
    console.log();
  }

  const result = await cypress.run({
    spec: specs,
    browser: 'chrome',
    headless: true,
    testingType: 'component',
  });

  const failedTests = result.runs.reduce((failed, { tests, spec }) => {
    return [...failed, ...tests.filter(({ state }) => state === 'failed').map((test) => ({ ...test, spec }))];
  }, []);

  specs = failedTests
    .map(({ spec }) => spec.relative)
    .reduce((result, spec) => {
      if (!result.includes(spec)) result.push(spec);
      return result;
    }, [])
    .join(',');

  if (failedTests.length === 0) {
    shouldRetry = false;
  } else if (retries >= maxRetries) {
    shouldRetry = false;
    shouldFail = true;
  } else {
    retries++;
  }
}

if (shouldFail) {
  console.log(chalk.bgRed("Couldn't recover these failing tests:"));
  console.log(
    chalk.red(
      specs
        .split(',')
        .map((spec) => `- ${spec}`)
        .join('\n')
    )
  );
  process.exit(1);
} else {
  console.log(chalk[retries === 0 ? 'bgBlue' : 'bgRed'](`Finished running tests with ${retries} retries`));
}

Under the same path we also have another script which collects paths of all our component test specs and "groups" them randomly, it looks like this:

#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const glob = require('glob');

const groupCount = process.argv[2];

if (groupCount === undefined) {
  console.error('Please provide a group count parameter');
  process.exit(1);
}

glob('src/**/*.cytest.tsx', (err, files) => {
  if (err) {
    console.error('An error occurred:', err);
    process.exit(1);
  }

  console.log('Found', files.length, 'files');

  // sort files randomly
  files.sort(() => Math.random() - 0.5);

  // take 1/3 of the files and put them in the first group to warm up Vite's cache
  const firstGroupAmount = Math.ceil(files.length / groupCount / 3);
  const groups = [files.splice(0, firstGroupAmount)];

  // ignore the first group in this loop
  for (let i = 0; i < groupCount - 1; i++) {
    groups.push(files.slice((i * files.length) / (groupCount - 1), ((i + 1) * files.length) / (groupCount - 1)));
  }

  console.log(
    'Created',
    groups.length,
    'groups with following distribution:',
    groups.map((group) => group.length)
  );

  fs.writeFileSync(path.join(__dirname, '..', '..', 'component-test-groups.json'), JSON.stringify(groups, null, 2));
});

On CI we have a following setup of steps:

• Distribute component tests randomly
  • ./bin/vite-migration/distribute-component-tests.js 5
• Component tests [1]
  • ./bin/vite-migration/cypress-ct-runner.mjs 1
• Component tests [2-5]
  • ./bin/vite-migration/cypress-ct-runner.mjs 2-5

Our whole CI flow looks like this:

I hope that someone will find this useful until the root cause gets fixed.

[lmiller1990]

Wow, what a hack - the fact you needed to do that really isn't ideal, I hope we can isolate and fix this soon. I wonder if we need to reach out to the Vite team - they'd probably have more insight than we would for the Vite internals.

Seems a lot of similar issues in Vite: https://github.com/vitejs/vite/issues?q=is%3Aissue+Failed+to+fetch+dynamically+imported+module

• https://github.com/vitejs/vite/issues/8715
• https://github.com/vitejs/vite/issues/11804 <= I posted in here, seems most recent

FYI we do the import here: https://github.com/cypress-io/cypress/blob/a7b290654e0cf87ea8ac94857d621a2d8ce633b2/npm/vite-dev-server/client/initCypressTests.js#L28

I wonder if we can add some pre-optimzation logic during CI mode to force Vite to pre-compile all dependencies, side stepping this issue entirely (which is sort of what the workarounds here are doing).

[lmiller1990]

Does anyone know any large OSS React + Vite + MUI projects I could try using to reproduce? I tried moving MUI core to Vite but it's not straight forward.

[matanAlltra]

This seems like a duplication of this thread so writing here also:

I have tried to run the tests: locally, on our Github Actions CI, cypress IO cloud and currents.dev(cypress cloud competitor). Out project is using React + Typescript.

locally: env: Macbook Pro 2021 12.5.1 (21G83) (M1 chip) running locally without docker result: failing after a few test runs

Github Actions: env: ubuntu-latest-4-cores docker: Github Worker Ubuntu 4 cores result: failing after a few test runs

Cypress.io env: unknown docker: unknown result: success - it seems to work as expected

currents.dev(cypress alternative) env: unknown docker: unknown result: success - it seems to work as expected

Haven't mention it until now but on the cloud solution it does seem to work. which leads me to the conclusion that it is something with my environment that I'm doing wrong

here is an example of a test I notice that fails more often

ApproveRun.spec.cy.tsx

import { ROUTES } from 'consts';
import { IN_PROGRESS } from 'consts/RUN';
import { MemoryRouter, Route, Routes } from 'react-router-dom';
import DynamicPage from 'routes/DynamicPage/DynamicPage';

import { beforeEachInit, run } from './util';

describe('Approve run', () => {
    beforeEach(() => {
        beforeEachInit();
        cy.wrappedMount(
            <MemoryRouter initialEntries={[`/run?runId=${run.id}`]}>
                <Routes>
                    <Route path="/:page" element={<DynamicPage />} />
                </Routes>
            </MemoryRouter>,
        );
    });

    it('should approve run action', () => {
        cy.getByTestId('ApprovalHeader__decline-button').click();

        cy.intercept(`${ROUTES.runRoute}/${run.id}`, {
            ok: true,
            run: { ...run, status: IN_PROGRESS },
        });

        cy.getByTestId('JsonFormModal_SubmitButton').click();

        cy.getByTestId('TagCell__turquoise').contains('In progress...');
    });
});

@lmiller1990 regarding the complexity of the test - I'd say it is pretty complex and heavy test because we render almost the top component in a SPA application regarding the imports - we are using absolute paths all across the codebase so it will be difficult to refactor. In our tests I did suspected that absolute paths imports may cause the issue so I tried to refactor the imports in the test to be relative instead of absolute but it didn't seem to solve the problem. It is pretty difficult to see if it actually helps tho because the error is not occurring in consistent way(not on a specific test).

@Murali-Puvvada the bash scripts is a workaround. tests that should take 300ms for example with the cypress run command takes 1000-3000ms instead because it rerun the chrome instance every time. Also: we don't have cypress e2e in our application(only component tests.)

I've uploaded the debug output to this public file(700mb log file) when I run all the tests: https://drive.google.com/file/d/1-2KOb6KV1SyOc_hBi2c2DK40gRtFeeop/view?usp=sharing

Would love any help regarding the issue.

here is the error I'm talking about:

[KaelWD]

We're seeing this in Vuetify's CI too: https://github.com/vuetifyjs/vuetify/actions/runs/4354367755/jobs/7609586657 It isn't always the same file, usually src/cypress/support/index.ts but I've seen spec files too like src/components/VInput/__tests__/VInput.spec.cy.tsx We're still on Vite 3.2.5, I'll try updating to 4.1.4 and see if that helps.

[matanAlltra]

@KaelWD I'm using Vite 4 and it doesn't seem to help

[chojnicki]

@lmiller1990 For us it's in docker and component testing. For months now... But what's weird:

It only happens on CI/CD (GH actions) where we using exact same docker container as on devs computers. On PC (docker) it works always. We have around 200 tests and in around 1/4 deploys it crashes on random test. It fails always on just single test, that is usually in the middle of list, so around 100th test. But not always. Sometimes it works days without any issue, and then fails multiple times per day. If I repeat workflow, it passes fully, but if it crashes again it is usually on same test. It's really annoying because it slows deployment heavily, If I have to repeat testing in CI/CD multiple times, when I know tests are just fine....

I was using Vite from beginning, and had this issue for months, trying multiple Cypress, browers and Vite versions and any workaround I could find. Everything is up to date.

It's hard to debug because like I said in never happens locally, just on GH. I was suspecting memory leak, but cypress/docker is running with stable ram usage, low for GH limits.

What I tried recently was configuring "attempts" to tests, so even if it fails, it should work fine second time. But this is useless option in that scenario, because when Failed to fetch dynamically imported module happens, Cypress just stops running that test and second attempt never happens........

Recently I added 2 packages that constantly optimized itself to optimizationDeps.entries. And I think that helped a little, but random failed tests still occurs.

Now I'm expirecing "failed to fetch" not component file, but cypress index file. Failed to fetch dynamically imported module: http://localhost:3000/__cypress/src/cypress/support/index.js Still on single test, and after rerun everything works.

[matanAlltra]

@chojnicki for me it's happening a lot of constantly. Currently the only workaround I've found was this bash script which run the tests separately one by one and if it fails than it retries for two times. Its significantly slower, but it does the job to only fail when a test actually fails. (we're running this bash script in GH actions too)

#!/bin/bash
ARGS=$@
all_failed=0

for file in $( find . -type f -name '*.spec.cy.tsx' ); do
    attempts=0
    while [ $attempts -lt 3 ]; do # Retry up to 3 times
        attempts=$((attempts+1))
        echo "Running $file (attempt $attempts)..."
        yarn cypress run $ARGS --component --browser chrome --spec $file && break
    done

    if [ $attempts -eq 3 ]; then # All attempts failed
        all_failed=1
        break
    fi
done

if [ $all_failed -eq 1 ]; then
    exit 1
fi

[lmiller1990]

We have an internal repro 🔥 FINALLY!

It's Cypress org private project, but dropping the link here so someone at Cypress can internally... https://cloud.cypress.io/projects/d9sdrd/runs/4193/test-results/2f924501-7fb2-4c80-b69f-819010c67c87

Now we've got a repro I can dig into it... for us it's CI only too, so sounds like a resources/race condition.

[lmiller1990]

Related: https://github.com/vitejs/vite/issues/11804

[matanAlltra]

FYI It seems I don't have a access to the link @lmiller1990

[chojnicki]

@lmiller1990 hope that repo you got will help, but if not, I think I could get access for you/cypress to our repo too.

[lmiller1990]

Hey team! Please add your planning poker estimate with Zenhub @astone123 @marktnoonan @mike-plummer @warrensplayer @jordanpowell88

[lmiller1990]

@chojnicki thanks a lot, I'll ping you if we need access, I think our reproduction should be enough. Is your reproduction consistent? CI only?

@matanAlltra sorry this reproduction repo is in our Cypress org but a private repo, I can't make it public it right now, but someone on our team will be able to see it and debug it. If anyone else can share a public repo with this issue, that would sure help, too!

[chojnicki]

@chojnicki thanks a lot, I'll ping you if we need access, I think our reproduction should be enough. Is your reproduction consistent? CI only?

@lmiller1990 like i described in previous comment - only CI/CD with GH actions. Locally (with exact same docker image, also on linux as host) it does not happening, not even once. But on GH it's very random.

[lmiller1990]

GH actions machines are notoriously under-resourced, that's why I was thinking it might be related. The reproduction I've got is ALSO GH actions - on the other hand, this repo has a couple hundred specs and runs on CircleCI and I've seen this happen there 0 times.

This is not the first GH actions specific issue related to timing/resources... but it's definitely useful to know GH actions is the best way to reproduce.

[andreikondratev]

We have the same issue with random "Failed to fetch dynamically imported module" in TeamCity agents running in google cloud without any parallelization. And yes these machines are also limited in resources.

[mschaefer-gresham]

For two weeks we have been using the suggestion of @synaptiko to run one small group of tests first before running the rest in parallel. We have not seen the error since doing this.

[lmiller1990]

Hi all - we are looking to work on this in the next two weeks with the goal of completely solving it, eliminating the need for work arounds.

[crobinson-expl]

@lmiller1990 I created a repro for this issue showing it happening in a NON-CI scenario. Hopefully, it helps get to the root of the problem.

git clone https://github.com/crobinson-expl/vue-scaffold-cypress-e2e-unit-march-15.git
cd vue-scaffold-cypress-e2e-unit-march-15
npm i
npm run test:unit:dev
// Choose Electron
// Click on HelloWorld.cy.ts

[lmiller1990]

Nice try @crobinson-expl but unfortunately your error is related to https://github.com/cypress-io/cypress/issues/26138, we don't support Vite 4.2 yet. That will be fixed in https://github.com/cypress-io/cypress/pull/26139.

The "failed to fetch" is so general, it can basically mean any kind of error occurred. I think this issue you are encountering is unrelated to the one reported here.

[waldemarennsaed]

Can we close this and mark it as fixed based on what we got in https://github.com/cypress-io/cypress/pull/26139 ?

[astone123]

@waldemarennsaed no I don't think so, since there are different problems described in this issue that don't relate to #26139. I would like to update the title of the issue to make it more specific though, since like Lachlan pointed out, the "failed to fetch" error can basically mean that any error occurred. I'll be looking more into this issue today

[lmiller1990]

@astone123 maybe useful: https://github.com/vitejs/vite/issues/11804#issuecomment-1464068181 could try using Vue's defineAsyncComponent to reproduce locally (maybe?) or the React version, which is React.lazy().

[astone123]

Okay we have a potential solution for this that should at least help prevent failures due to this issue. If anyone is willing, you can try installing a pre-release binary from my commit here.

If anyone does this, please post here and let us know if you are still seeing import failures in CI

[stnokott]

If anyone does this, please post here and let us know if you are still seeing import failures in CI

Before this change, 90% of my Cypress runs in the CI of my little hobby project were failing due to dynamic import issues. Since using your commit, none of them failed due to that reason. (It's a tiny project, so probably not very representative, but wanted to let you know regardless)

[waldemarennsaed]

This issue seems to be fixed for me by release 12.9.0 - I don't see the error message anymore and my component-tests succeed.

Appears to be fixed by/in: https://github.com/cypress-io/cypress/issues/26138

[chojnicki]

So I can confirm it was not fixed in my case because based on #26138 it was due to vite 4.2.0, but I'm still on 4.1.1 :/ I will update Cypress anyway with that fix, and wait until it will happen again.

[chojnicki]

Lol, I did not had to wait too long. Failed on first try after update cypress to 12.9.0.

✖ 1 of 191 failed (1%) 00:37 654 653 1 - -

Running:  landing/LandingArticleCard.spec.js                                          (109 of 191)

  (Attempt 1 of 2) An uncaught error was detected outside of a test
  1) An uncaught error was detected outside of a test

  0 passing (419ms)
  1 failing

  1) An uncaught error was detected outside of a test:
     TypeError: The following error originated from your test code, not from Cypress.

  > Failed to fetch dynamically imported module: http://localhost:3000/__cypress/src/cypress/support/index.js

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.

Cypress could not associate this error to any specific test.

We dynamically generated a new test to display this failure.

Again after around 50% finished tests, again random one, component is just fine (it's very basic) and it will work after rerun, it happens only on GH actions (works locally). So nothing changed :(

Like I said before, I tried with "retries" functionality in Cypress, but it ignores it. Like you see, there is (Attempt 1 of 2) but it never tries second time. I guess it only retries when there are assertion errors, but not general errors, which is shame :( I would really like to just rerun that single test, instead all queue and all CI/CD process. Any way to do that?

Edit: Also it's failing on importing cypress/support/index.js which for me does not make any sense.

[KaelWD]

@astone123 retry-dynamic-imports doesn't seem to help for me: https://github.com/vuetifyjs/vuetify/actions/runs/4554236749/jobs/8031835084

[lmiller1990]

If anyone does this, please post here and let us know if you are still seeing import failures in CI

Before this change, 90% of my Cypress runs in the CI of my little hobby project were failing due to dynamic import issues. Since using your commit, none of them failed due to that reason. (It's a tiny project, so probably not very representative, but wanted to let you know regardless)

Can you share your hobby project by any chance? Having an easy, reliable reproduction helps.

@KaelWD thanks for sharing that! How often do you see this?

This is driving me nuts! I'm sure there's something going on in Vite that needs patching (https://github.com/vitejs/vite/issues/11804), but I have no idea what. It's really difficult to debug this, given it only happens on CI, and only sometimes.

We ended up opting out of https://github.com/cypress-io/cypress/pull/26202, it didn't fix the error for our GH Actions project. It's not a great fix, either way. The fact this is GH Actions only is very suspicious! Is it really just resources related?

I wonder if we can compile and not use a dev-server at all for CI - would this sidestep the issue entirely 🤔

[stnokott]

Can you share your hobby project by any chance? Having an easy, reliable reproduction helps.

https://github.com/stnokott/r6-dissect-influx Change was introduced in https://github.com/stnokott/r6-dissect-influx/pull/82

(note that some PRs are currently failing, but due to a different issue related to Typescript 5)

[KaelWD]

How often do you see this?

Looks like around 50% of the time. It started when I updated cypress from 10.2.0 to 12.5.1 so I can't really narrow it down to a specific version.

[lmiller1990]

@stnokott thanks for this - the TS issue is, in fact, a bug in TypeScript. Follow this issue: https://github.com/cypress-io/cypress/issues/26148#issuecomment-1487574243. It looks like there is a open PR in TS fixing the issue. 🤞 Looking at your CI, it seems the same test, Settings.cy.tsx keeps failing, and the Vite optimize step always precedds it? https://github.com/stnokott/r6-dissect-influx/actions/runs/4467180776/jobs/7846310109#step:5:271. I wonder if you add those deps to optimizeDeps in your vite.config, will it prebundle them and avoid the flake 🤔 ?

@KaelWD thanks, I will spend some more time looking into this today.

[lmiller1990]

Just collecting research for myself to debug this, if it is a slow start caching issue, maybe we can adapt https://github.com/bluwy/vite-plugin-warmup

Edit @KaelWD interesting, I got a failure on first try for Vuetify. I added the plugin above, and it passed four times consecutively. https://github.com/lmiller1990/vuetify/actions. WDYT? Can you try adding that plugin to your CI like I did and see if it helps?

Edit: I removed the plugin and it failed. Either this was dumb luck, or we isolated the issue to a race condition, solved by having a warm cache.

@stnokott can you try the plugin too? And @chojnicki? Like this: https://github.com/lmiller1990/vuetify/commit/ff65846fcea3461092e908aa60cc94d07aa83102#diff-ee0cbc5388a0d00ac00c818b7cf76db70ff9c4e7945808ea2340d4eb4423d4cfR58-R63 include a pattern to match your spec files, and your cypress/component file (the one with mount).

[stnokott]

@stnokott can you try the plugin too? And @chojnicki? Like this: lmiller1990/vuetify@ff65846#diff-ee0cbc5388a0d00ac00c818b7cf76db70ff9c4e7945808ea2340d4eb4423d4cfR58-R63 include a pattern to match your spec files, and your cypress/component file (the one with mount).

Yup, works with the plugin instead of the workaround, but had to force the installation due to dependency issues (see https://github.com/stnokott/r6-dissect-influx/actions/runs/4573602960/jobs/8074257398). Probably not an issue anymore once the plugin doesn't depend on a beta of Vite anymore.

But yeah, looks promising, also makes my tests run faster in general.

[lmiller1990]

@stnokott Great to hear! If enough people try the plugin and it turns out this is the solution, we will build it (or something similar, probably with some Cypress specific tweaks, like auto-adding supportFile and finding your specs etc) into our Vite Dev Server, so you don't need to install another dependency.

[KaelWD]

Sample size of 1 but seems to work so far: https://github.com/vuetifyjs/vuetify/actions/runs/4574802124 Looks like evan is interested in having it be part of vite itself: https://twitter.com/youyuxi/status/1641402565631025152

Aaand still busted: https://github.com/vuetifyjs/vuetify/actions/runs/4576029601/jobs/8079626308 unless process.env.CYPRESS isn't set somehow but it seems to be locally

[lmiller1990]

Argh, the dream was short lived. It's hard to say if this made a difference or if I just got lucky with 4 consecutive passes. 😢

Hmm you are running 4 containers - I'll try running more than 1, maybe that's related.

[chojnicki]

@lmiller1990 we had so far multiple builds without failing, until now. But it's at least different error, so progress :D

Never saw this one before. After return everything worked. So I guess I disable retries (since it does not work anyway like I wanted, what I described before), but not sure if it's really related to that option.

  Running:  contact/ContactTeamSection.spec.js                                           (55 of 194)

6:50:21 AM [vite] ✨ new dependencies optimized: @sentry/vue, @sentry/tracing, nprogress, floating-vue
6:50:21 AM [vite] ✨ optimized dependencies changed. reloading
  ✓ renders title (62ms)
  ✓ renders subtitle when specified (31ms)
  (Attempt 1 of 2) renders content specified in slot
  1) renders content specified in slot

  2 passing (515ms)
  1 failing

  1) renders content specified in slot:
     TypeError: Cannot read properties of undefined (reading 'prevAttempts')
      at replacePreviousAttemptWith (http://localhost:3000/__cypress/runner/cypress_runner.js:158093:40)
      at Object.onRunnableRun (http://localhost:3000/__cypress/runner/cypress_runner.js:158232:11)
      at $Cypress.action (http://localhost:3000/__cypress/runner/cypress_runner.js:147632:28)
      at Runnable.run (http://localhost:3000/__cypress/runner/cypress_runner.js:156392:13)
      at ../driver/node_modules/mocha/lib/runner.js.Runner.runTest (http://localhost:3000/__cypress/runner/cypress_runner.js:105623:10)
      at http://localhost:3000/__cypress/runner/cypress_runner.js:105749:12
      at next (http://localhost:3000/__cypress/runner/cypress_runner.js:105532:14)
      at http://localhost:3000/__cypress/runner/cypress_runner.js:105542:7
      at next (http://localhost:3000/__cypress/runner/cypress_runner.js:105444:14)
      at http://localhost:3000/__cypress/runner/cypress_runner.js:105510:5
      at timeslice (http://localhost:3000/__cypress/runner/cypress_runner.js:99436:27)

[lmiller1990]

@chojnicki you used the hot caching plugin? I wonder why it still does

6:50:21 AM [vite] ✨ new dependencies optimized: @sentry/vue, @sentry/tracing, nprogress, floating-vue

The hot plugin should have found those.

Can you trying adding those to optimizeDeps.include? https://vitejs.dev/config/dep-optimization-options.html#optimizedeps-include. There is also optimizeDeps.entries, but I don't know if that works for node_modules (seems not, according to the docs).