when builder is symlinked, environment causes TypeScript compiler to fork bomb

[trusktr]

For some reason, when builder is symlinked (for example, Lerna cross links projects in mono- or umbrella- repos using symlinks) then for some reason this causes a simple builder script that runs tsc to fork bomb and eventually crash the whole system (I need to reboot if I don't quit out of the CLI before it crashes).

The script I have in my archetype is simple, like this:

        "tsc": "tsc -p ./tsconfig.json",

This archetype, and builder itself, are both symlinked into the consuming project.

In the project, when I run LOCAL_DEV=true builder run tsc, then builder spawns (fork bombs?) a bunch of tsc processes indefinitely until the OS crashes.

If I try to manually run the tsc script manually by running

./node_modules/builder-js-package/node_modules/typescript/bin/tsc -p ./tsconfig.json

where builder-js-package is my symlinked archetype, then the process works just fine, and tsc completes a build.

So the indefinite spawning (or fork bomb) happens only when I try to run the tsc command via a builder script.

Reproduction:

$ git clone --recursive git@github.com:infamous/umbrella.git
... output omitted ...
$ cd umbrella
$ git checkout builder-issue-182
$ npx lerna bootstrap # this links projects in apps/ and packages/ to each other as needed
... output omitted ...
$ cd packages/lowclass
$ ls -ld node_modules/builder* # this shows that builder and archetype are symlinked
lrwxrwxrwx 1 trusktr trusktr 28 Feb 16 09:50 node_modules/builder -> ../../FormidableLabs+builder/
lrwxrwxrwx 1 trusktr trusktr 24 Feb 16 09:50 node_modules/builder-js-package -> ../../builder-js-package/
$ npm run build

> lowclass@4.9.2 build /home/trusktr/src/lume+umbrella/packages/lowclass
> LOCAL_DEV=true builder run build:prod

┬  ┌─┐┬ ┬┌─┐┬  ┌─┐┌─┐┌─┐
│  │ │││││  │  ├─┤└─┐└─┐
┴─┘└─┘└┴┘└─┘┴─┘┴ ┴└─┘└─┘

Note that all builder commands are prefixed with LOCAL_DEV=true, otherwise the symlinks cause builder not to find any package.json files, and thus it can't begin to run any scripts.

After that last bit of output, it will be stuck continuously spawning new processes.

While it is doing that, from another terminal you can run the following repeatedly to see the number of processes growing over time (which thankfully is not happening so rapidly and the system will be responsive for a while instead of crashing right away):

ps -U <USERNAME> | wc -l

Replace <USERNAME> with your username.

For example, this is what I see in my terminal:

~ ❯ ps -U trusktr | wc -l
134

~ ❯ ps -U trusktr | wc -l
137

~ ❯ ps -U trusktr | wc -l
140

~ ❯ ps -U trusktr | wc -l
143

~ ❯ ps -U trusktr | wc -l
146

~ ❯ ps -U trusktr | wc -l
149

~ ❯ ps -U trusktr | wc -l
152

~ ❯ ps -U trusktr | wc -l
156

~ ❯ ps -U trusktr | wc -l
159

~ ❯ ps -U trusktr | wc -l
162

You can also list all of these processes:

~ ❯ ps aux | grep tsc
trusktr  14263  5.0  0.2 586448 42888 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14270  5.2  0.2 586444 43072 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14277  5.5  0.2 586552 43304 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14284  5.2  0.2 586452 43148 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14291  5.5  0.2 586500 43648 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14299  7.3  0.2 586720 43148 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14306  6.6  0.2 586684 43232 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14313  6.6  0.2 586464 43356 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14321  7.0  0.2 586424 42940 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14330  8.0  0.2 586688 43084 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
trusktr  14348 13.0  0.2 586436 43368 pts/2    Sl+  13:08   0:00 node /home/trusktr/src/lume+umbrella/packages/lowclass/node_modules/.bin/builder run tsc
... output omitted ...

and the list will get longer and longer by a few processes every second. It is growing by 3-5 processes or more per second (depending on your CPU speed I imagine).

[ryan-roemer]

LOCAL_DEV isn't meant to be used as a normal thing. Symlinks are going to be problematic for many things with builder. Your best bet is to probably look into https://github.com/FormidableLabs/lank#readme and see if there's a lank config that can get everything to work (note that it will conflict with symlink approach if same package at issue). Good luck!

[trusktr]

Interesting, thanks for sharing lank. It's interesting that it takes a different approach than symlinking; I haven't seen any other tools take that approach.

Another problem is that even though LOCAL_DEV is set, it seems to work only with the main archetype, but the variable does not have any effect on resolution of *-dev archetypes so all the same problems persist in any project using a -dev archetype regardless of that variable. I.e., the same problems as main archetypes without LOCAL_DEV being set, happen with -dev archetypes regardless of that variable being set or not.

---

For now, my new workaround is to simply make a package that provides a bin, and using commander to set up command like my-cli build and my-cli test, etc, that simply call functions with the equivalent names exported from a config file overridable/extendable by each project. This does not rely on mangling PATH or any env variables, its just runtime functions.

Regardless of this workaround for executing common tooling across projects, I think lank might be useful. I'll have to try that to see how it compares with other tools that symlink. Do you think the lack of symlinks will help builder operate as usual? If so, then maybe I don't need to implement my workaround.

[trusktr]

Oops, I forgot to push the reproduction branch I mentioned in the OP, so I pushed it up in case someone ever wants to try to reproduce it.