Files in `exclude` are still processed by the build system, leading to hangs

LegNeato commented 4 days ago

Have you read the Contributing Guidelines on issues?

[X] I have read the Contributing Guidelines on issues.

Prerequisites

[X] I'm using the latest version of Docusaurus.
[X] I have tried the npm run clear or yarn clear command.
[X] I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
[X] I have tried creating a repro with https://new.docusaurus.io.
[X] I have read the console error message carefully (if applicable).

Description

I have a site that uses the classic preset. There is some Rust code in one of the blog directories (blog/2024-11-21-optimizing-matrix-mul/code). I explicitly exclude the Rust code and subdirectories from the blog config:

Snippet: ```ts presets: [ [ "classic", { blog: { exclude: ["*/code/**"], ``` [Full config is on GitHub](https://github.com/Rust-GPU/rust-gpu.github.io/blob/main/docusaurus.config.ts)

Rust puts the target/ output directory with intermediate and compiled artifacts in the top-level (that is, blog/2024-11-21-optimizing-matrix-mul/code/target). After doing a couple of Rust builds (cargo build) and benchmark runs (cargo bench) the target directory is full of many files, some large.

With a full target/ directory, docusaurus stops building the website successfully. Both yarn start and yarn build make some progress and then stall out with node using 100% CPU. I have waited for 20 mins with no forward progress (usually builds are < 10s). Deleting the blog/2024-11-21-optimizing-matrix-mul/code/target directory's many files enables the build to make forward progress and succeed.

Note when I run yarn start and then do a Rust build in the supposedly excluded directory, I can see the client hot-reloading / compiling being kicked off as well.

I know the exclude entry in the config is being respected, as previously I had to put a truncate marker in README.md files in the Rust code (I have missing markers set to throw). After I put the exclude entry in the settings, markdown files in code were correctly not being treated as blog posts and I could remove the truncate markers.

So, it appears that something in the build process is globbing or reading/processing each file and then applying the excludes.

Reproducible demo

https://github.com/Rust-GPU/rust-gpu.github.io

Steps to reproduce

Exclude files in some path in the blog plugin
Run yarn start
touch a file in the excude path
See that a build is triggered

Expected behavior

Build completes.

Actual behavior

Build is triggered. If there are tons of files, node goes to 100% CPU usage during yarn build, no progress on the build. Deleting the files in "exclude" directory allows the build to make progress and finish.

Your environment

Public source code: https://github.com/Rust-GPU/rust-gpu.github.io
Public site URL: rust-gpu.github.io
Docusaurus version used: tried 3.5 and 3.6.2 (with and without build fast, with and without all the sub build fast options turned on)
Environment name and version (e.g. Chrome 89, Node.js 16.4): Node v23.2.0
Operating system and version (e.g. Ubuntu 20.04.2 LTS): macoS sequoia

Self-service

[ ] I'd be willing to fix this bug myself.

slorber commented 3 days ago

Can you provide a full repro branch where I can reliably reproduce the problem?

If cargo generates files, then please commit them.

slorber commented 3 days ago

Your exclude overrides the default, maybe the problem could be related:

  include: ['**/*.{md,mdx}'],
  exclude: [
      '**/_*.{js,jsx,ts,tsx,md,mdx}',
      '**/_*/**',
      '**/*.test.{js,jsx,ts,tsx}',
      '**/__tests__/**',
  ]

Maybe try appending to that exclude list instead.

And also try with ** instead of *:

  exclude: [
      '**/_*.{js,jsx,ts,tsx,md,mdx}',
      '**/_*/**',
      '**/*.test.{js,jsx,ts,tsx}',
      '**/__tests__/**',
      '**/code/**',
  ]