Setting manualChunks breaks react lazy loading

Describe the bug

I'm using a react component like

import { lazy, Suspense } from "react";
const Lazy = lazy(() => import("./LazyCanvasPreload"));

const FullPageSpinner = () => <>spin...</>;

const Component = () => {
  return (
    <>
      <Suspense fallback={<FullPageSpinner />}>
        <Lazy />
      </Suspense>
    </>
  );
};

export default Component;

to load a heavy three.js canvas. Without lazy loading this component makes the first paint have to wait for the ~2MB three and rapier libraries.

Vite does successfully chunk this component and it works great. Until I add this vite config:

{
    plugins,
    build: {
      rollupOptions: {
        output: {
          manualChunks(id: string) {
            const module = id.split("/node_modules/")[1];
            if (!module) {
              return; // void
            }
            if (module.match(/^@react-three\/drei\//)) return "drei";
            if (module.match(/^@react-three\/fiber\//)) return "fiber";
            if (module.match(/^@react-three\/rapier\//)) return "rapier";
            if (module.match(/^leva\//)) return "leva";
            if (module.match(/^meshline\//)) return "meshline";
            if (module.match(/^three\//)) return "three";
            return; // void
          },
        },
      },
    },
  }

Now my vendor libraries are correctly split but, but all of the sudden my first paint is now dependent on loading the three (but not the rapier) chunks making first paint incredibly slow. I'm using the vercel 3d tag as an example with asset preloading. The first paint is delayed by 18,000ms in chrome's slow 3G preset

I did try returning a string instead of void in all non-matched returns without an improvement.

screenshots: https://imgur.com/a/BbfO8NT

Reproduction

https://github.com/yspreen/vite-bug-showcase

deployed:
https://vite-bug-no-manual-chunks.pages.dev
https://vite-bug-manual-chunks.pages.dev

Steps to reproduce

install, build and serve both variants of the showcase project.

one of them will show the spinner immediately, the other one only shows the spinner after loading the three.js library

see screenshots: https://imgur.com/a/BbfO8NT

System Info

macOS, latest Chrome

Used Package Manager

yarn

Logs

vite v5.3.3 building for production...

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

vitejs / vite