search

google / wireit

Wireit upgrades your npm/pnpm/yarn scripts to make them smarter and more efficient.
Apache License 2.0
6.01k stars 105 forks source link

[bug report] Wireit watch crashes recursively looping symbolic links in a npm workspaces context. #790

Open vdegenne opened 1 year ago

vdegenne commented 1 year ago

Describe the bug

It seems that wireit follows symbolic links when using the --watch option, it would work fine in most of cases but there is a rare one where if a project in a monorepo (npm workspaces) depends on the root project then it will keep looping inside the node_modules directory. Here's a screenshot of what could possibly happens : image

To Reproduce

Best way to reproduce that is to actually try it in a real case scenario:

You should see the problem.

Priority

This is probably a rare case but right now it's impossible for some users to contribute to the material-web catalog repository because of this issue. Could happen to any monorepo project that set the root as one workspace (e.g. "workspaces": [".", "foo"])

Version

wireit : 0.9.0

e111077 commented 1 year ago

On Mac there seems to be a memory leak that continues to build with no activity and the --watch command. After a while I get an eventual heap size crash:

<--- Last few GCs --->

[14155:0x140008000]  2227414 ms: Scavenge 3985.5 (4128.1) -> 3985.1 (4128.1) MB, 5.2 / 0.0 ms  (average mu = 0.062, current mu = 0.028) external memory pressure; 
[14155:0x140008000]  2227563 ms: Scavenge 3997.7 (4128.1) -> 3985.8 (4128.1) MB, 6.6 / 0.0 ms  (average mu = 0.062, current mu = 0.028) task; 
[14155:0x140008000]  2227576 ms: Scavenge 3989.4 (4128.1) -> 3986.9 (4128.4) MB, 6.2 / 0.0 ms  (average mu = 0.062, current mu = 0.028) external memory pressure; 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100632180 node::Abort() [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 2: 0x100632374 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 3: 0x10078fc6c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 4: 0x100950028 v8::internal::EmbedderStackStateScope::EmbedderStackStateScope(v8::internal::Heap*, v8::internal::EmbedderStackStateScope::Origin, cppgc::EmbedderStackState) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 5: 0x100954318 v8::internal::Heap::CollectGarbageShared(v8::internal::LocalHeap*, v8::internal::GarbageCollectionReason) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 6: 0x100950b44 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 7: 0x10094e1c0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 8: 0x10094d650 v8::internal::Heap::HandleGCRequest() [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
 9: 0x1008f1e98 v8::internal::StackGuard::HandleInterrupts() [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
10: 0x100ce6494 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
11: 0x1010772ac Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
12: 0x101026698 Builtins_ArrayIteratorPrototypeNext [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
13: 0x1010c4900 Builtins_PromiseAll [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
14: 0x10605b854 
15: 0x10102a8b4 Builtins_AsyncFunctionAwaitResolveClosure [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
16: 0x1010cae38 Builtins_PromiseFulfillReactionJob [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
17: 0x10101a834 Builtins_RunMicrotasks [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
18: 0x100ff23c4 Builtins_JSRunMicrotasksEntry [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
19: 0x1008c7dcc v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
20: 0x1008c82bc v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
21: 0x1008c8498 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
22: 0x1008f0410 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
23: 0x1008f0cac v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
24: 0x100580c5c node::InternalCallbackScope::Close() [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
25: 0x100580fe8 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
26: 0x100596264 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
27: 0x100637948 node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
28: 0x1006393f0 node::fs::AfterScanDir(uv_fs_s*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
29: 0x10062e988 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
30: 0x100fce80c uv__work_done [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
31: 0x100fd1fb4 uv__async_io [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
32: 0x100fe4758 uv__io_poll [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
33: 0x100fd2484 uv_run [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
34: 0x100581704 node::SpinEventLoopInternal(node::Environment*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
35: 0x10066f9e4 node::NodeMainInstance::Run() [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
36: 0x1005fe194 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
37: 0x1005fe3ec node::Start(int, char**) [/Users/emarquez/.nvm/versions/node/v19.6.0/bin/node]
38: 0x198a0ff28 start [/usr/lib/dyld]
[1]    14096 abort      npm run dev
peschee commented 7 months ago

Are there any updates on this?

e111077 commented 7 months ago

we were able to fix this by being more careful on not watching node_modules:

https://github.com/material-components/material-web/pull/4509

Can also confirm that we do not run into this problem at all on Mac anymore

peschee commented 7 months ago

@e111077 Thanks, I've tweaked a glob pattern and it seems it has gotten better. I will keep an eye out whether this fixes things for us.

Initially, I thought I can write a small script that checks all globs for files in all wireit tasks in all package.json files in our monorepo by detecting whether the glob patterns might potentially include things from node_modules (and hence cause issues). However, I noticed that I cannot just purely rely on fast-glob (which is what wireit uses internally) because it seems wireit wraps fast-glob with some additional logic: https://github.com/google/wireit/blob/v0.14.4/src/util/glob.ts#L117

To make things even more complicated, there's this TODO comment from @aomarks: https://github.com/google/wireit/blob/v0.14.4/src/watcher.ts#L484 which mentions issue https://github.com/google/wireit/issues/550

It would be nice if some sort of verbose/debug mode existed for things related to files and output, since things can get messy with glob patterns, and we cannot use the ignore option in fast-glob (https://github.com/mrmlnc/fast-glob?tab=readme-ov-file#ignore) which would simplify these potentially complex glob patterns significantly, when all you want is to exclude node_modules from everywhere for example.