search

mrmlnc / fast-glob

:rocket: It's a very fast and efficient glob library for Node.js
MIT License
2.56k stars 112 forks source link

Bug since v2.0.0 — RangeError: Maximum call stack size exceeded #42

Closed maxmilton closed 6 years ago

maxmilton commented 6 years ago

Environment

Actual behavior

Since v2.0.0 an error is thrown when run on a filesystem. Number of files/dirs is 54/18 in my case.

Call stack:

yarn run v1.3.2
$ OVERRIDE_ENV=development npm run build

> app@0.0.0 build /home/max/Development/example/packages/app
> APP_RELEASE=$(git describe --always) NODE_ENV=production node build/build.js

/home/max/Development/example/node_modules/fast-glob/out/providers/reader-sync.js:40
            throw err;
            ^

RangeError: Maximum call stack size exceeded
    at call.safe (/home/max/Development/example/node_modules/readdir-enhanced/lib/stat.js:26:26)
    at onceWrapper (/home/max/Development/example/node_modules/readdir-enhanced/lib/call.js:45:17)
    at onceWrapper (/home/max/Development/example/node_modules/readdir-enhanced/lib/call.js:45:17)
    at exports.lstat (/home/max/Development/example/node_modules/readdir-enhanced/lib/sync/fs.js:59:5)
    at Object.safeCall [as safe] (/home/max/Development/example/node_modules/readdir-enhanced/lib/call.js:24:8)
    at stat (/home/max/Development/example/node_modules/readdir-enhanced/lib/stat.js:19:8)
    at DirectoryReader.processItem (/home/max/Development/example/node_modules/readdir-enhanced/lib/directory-reader.js:171:5)
    at array.forEach.item (/home/max/Development/example/node_modules/readdir-enhanced/lib/sync/for-each.js:14:5)
    at Array.forEach (<anonymous>)
    at Object.syncForEach [as forEach] (/home/max/Development/example/node_modules/readdir-enhanced/lib/sync/for-each.js:13:9)
    at call.safe (/home/max/Development/example/node_modules/readdir-enhanced/lib/directory-reader.js:80:16)
    at onceWrapper (/home/max/Development/example/node_modules/readdir-enhanced/lib/call.js:45:17)
    at onceWrapper (/home/max/Development/example/node_modules/readdir-enhanced/lib/call.js:45:17)
    at exports.readdir (/home/max/Development/example/node_modules/readdir-enhanced/lib/sync/fs.js:19:5)
    at Object.safeCall [as safe] (/home/max/Development/example/node_modules/readdir-enhanced/lib/call.js:24:8)
    at DirectoryReader.readNextDirectory (/home/max/Development/example/node_modules/readdir-enhanced/lib/directory-reader.js:71:10)

npm ERR! Linux 4.14.14-300.fc27.x86_64
npm ERR! argv "/usr/bin/node" "/home/max/.config/yarn/global/node_modules/.bin/npm" "run" "build"
npm ERR! node v9.4.0
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! app@0.0.0 build: `APP_RELEASE=$(git describe --always) NODE_ENV=production node build/build.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the app@0.0.0 build script 'APP_RELEASE=$(git describe --always) NODE_ENV=production node build/build.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the app package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     APP_RELEASE=$(git describe --always) NODE_ENV=production node build/build.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs app
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls app
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/max/Development/example/packages/app/npm-debug.log
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Looks to be related to readdir-enhanced.

Expected behavior

fast-glob runs without any error thrown.

Steps to reproduce

  1. Nothing specific, just run fast-glob.

Code sample

const fastGlob = require('fast-glob');

const paths = fastGlob.sync([
  path.join(__dirname, './**/*.html'),
  path.join(__dirname, './**/*.js'),
]);

NOTE

Since v2.0.0 has removed bashNative mode, I'm actually going to pin my fast-glob version to 1.0.1 anyway since I want that native level of performance. That was actually the reason why I started using fast-glob in the first place!

mrmlnc commented 6 years ago

Hello, @MaxMilton,

You can provide find . | wc -l and tree -L 1 output for your __dirname directory? I think you have a very large project directory. Also fast-glob@2now faster then bash-glob (it is possible that not in all cases).

maxmilton commented 6 years ago

The number of files in my project root is large, total dirs+files = 27405. Also the tree:

.
├── build
├── config
├── dist
├── index.html
├── jsconfig.json
├── node_modules
├── package.json
├── src
├── static
├── test
└── yarn.lock

7 directories, 4 files

Does fast-glob search the entire project from the __dirname? My specific use case is similar to this:

build/conf.js:

fastGlob.sync([
  path.join(__dirname, '../index.html'),
  path.join(__dirname, '../src/**/*.vue'),
  path.join(__dirname, '../src/**/*.js'),
]),

That's good to hear v2 is faster, obviously I haven't been able to test.

mrmlnc commented 6 years ago

Thanks for provided details.

I'll work on your problem tonight. Now you can exclude node_modules directory from reading by !**/node_modules pattern. Also you can exclude dest and other not needed directories.

Sorry for this situation.

maxmilton commented 6 years ago

Using your recommended excludes I'm able to run without any errors. For anyone else's reference, my updated and working code is:

fastGlob.sync([
  '!**/node_modules',
  '!**/dist',
  path.join(__dirname, '../index.html'),
  path.join(__dirname, '../src/**/*.vue'),
  path.join(__dirname, '../src/**/*.js'),
]),

This is fine for me so feel free to close this issue.

maxmilton commented 6 years ago

After reading the documentation more closely I realised fast-glob starts the search from process.cwd() so I was able to simplify things:

fastGlob.sync([
  '!node_modules',
  '!dist',
  'index.html',
  'src/**/*.vue',
  'src/**/*.js',
]),

Thank you for your hard work on this project :1st_place_medal:

mrmlnc commented 6 years ago

Well, I need to understand what's going on.

If you have free time, then tell me how much free memory you have on the server?

I checked work of this the package on directories with more than 500 000 files and it just works. But I have 16GB RAM and a heap of V8 has a size of 1.4 GB.

Maybe I can use a more intelligent index for occurrences or try to exclude the directories like .git, node_modules by default. So...

So, it's just a working theory. Maybe i'm wrong.

mrmlnc commented 6 years ago

So, we well work on the solution here:

mrmlnc commented 6 years ago

@MaxMilton,

Please, try to use npm i fast-glob@next --force for your broken cases. I tried to fix the problem with large directories, temporarily switching to the readdir-enhanced fork.

maxmilton commented 6 years ago

Sorry I hadn't gotten back to this thread sooner. I can confirm on the current v2.0.4 the issue I originally raised is no long reproducible.

Thank you for your hard work on this project! :star2: