search

fedimint / ui

https://ui-umber-ten.vercel.app
MIT License
26 stars 39 forks source link

Error: EMFILE: too many open files, watch '.' - Ubuntu 22.04 #370

Closed EthnTuttle closed 6 months ago

EthnTuttle commented 7 months ago

I am unable to run the dev environment for guardian or gateway due to the below error. It is the same for either app.

$ turbo run dev --filter=guardian-ui...
╭───────────────────────────────────────────────────────────────────────╮
│                                                                       │
│                   Update available v1.9.3 ≫ v1.12.3                   │
│    Changelog: https://github.com/vercel/turbo/releases/tag/v1.12.3    │
│               Run "npx @turbo/codemod update" to update               │
│                                                                       │
│     Follow @turborepo for updates: https://twitter.com/turborepo      │
╰───────────────────────────────────────────────────────────────────────╯
 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please 
ensure the following:
        - the process identified by the pid in the file at /tmp/nix-shell.UnxQ78/turbod/5495566afd0848ec/turbod.pid i
s not running, and remove /tmp/nix-shell.UnxQ78/turbod/5495566afd0848ec/turbod.pid
        - check the logs at /home/ethan/.local/share/turborepo/logs/5495566afd0848ec-ui.log
        - the unix domain socket at /tmp/nix-shell.UnxQ78/turbod/5495566afd0848ec/turbod.sock has been removed
        You can also run without the daemon process by passing --no-daemon
• Packages in scope: @fedimint/eslint-config, @fedimint/guardian-ui, @fedimint/tsconfig, @fedimint/types, @fedimint/u
i, @fedimint/utils
• Running dev in 6 packages
• Remote caching disabled
@fedimint/types:dev: cache bypass, force executing b8634f9a316f84ee
@fedimint/types:build: cache hit, replaying output 2f17b030faa71f38
@fedimint/types:build: $ tsup
@fedimint/types:build: CLI Building entry: src/index.ts
@fedimint/types:build: CLI Using tsconfig: tsconfig.json
@fedimint/types:build: CLI tsup v5.12.9
@fedimint/types:build: CLI Using tsup config: /home/ethan/code/ui/packages/types/tsup.config.json
@fedimint/types:build: CLI Target: node12
@fedimint/types:build: CJS Build start
@fedimint/types:build: CJS ⚡️ Build success in 41ms
@fedimint/types:build: CJS dist/index.js     2.54 KB
@fedimint/types:build: CJS dist/index.js.map 6.30 KB
@fedimint/types:build: DTS Build start
@fedimint/types:build: DTS ⚡️ Build success in 2074ms
@fedimint/types:build: DTS dist/index.d.ts 6.15 KB
@fedimint/utils:dev: cache bypass, force executing 3b93cd9fb8a2be9b
@fedimint/utils:build: cache hit, replaying output a8e4ebdbfcf28fa0
@fedimint/utils:build: $ yarn run tsup
@fedimint/utils:build: $ /home/ethan/code/ui/node_modules/.bin/tsup
@fedimint/utils:build: CLI Building entry: src/index.tsx
@fedimint/utils:build: CLI Using tsconfig: tsconfig.json
@fedimint/utils:build: CLI tsup v5.12.9
@fedimint/utils:build: CLI Using tsup config: /home/ethan/code/ui/packages/utils/tsup.config.json
@fedimint/utils:build: CLI Target: node12
@fedimint/utils:build: CJS Build start
@fedimint/utils:build: CJS ⚡️ Build success in 20ms
@fedimint/utils:build: CJS dist/index.js     2.64 KB
@fedimint/utils:build: CJS dist/index.js.map 2.36 KB
@fedimint/utils:build: DTS Build start
@fedimint/utils:build: DTS ⚡️ Build success in 2032ms
@fedimint/utils:build: DTS dist/index.d.ts 751.00 B
@fedimint/ui:dev: cache bypass, force executing 04faf5660470ce6a
@fedimint/ui:build: cache hit, replaying output d32c21476bafe57e
@fedimint/ui:build: $ tsup
@fedimint/ui:build: CLI Building entry: src/index.tsx
@fedimint/ui:build: CLI Using tsconfig: tsconfig.json
@fedimint/ui:build: CLI tsup v5.12.9
@fedimint/ui:build: CLI Using tsup config: /home/ethan/code/ui/packages/ui/tsup.config.json
@fedimint/ui:build: CLI Target: node12
@fedimint/ui:build: CJS Build start
@fedimint/ui:build: CJS ⚡️ Build success in 37ms
@fedimint/ui:build: CJS dist/index.js     39.12 KB
@fedimint/ui:build: CJS dist/index.js.map 55.42 KB
@fedimint/ui:build: DTS Build start
@fedimint/ui:build: DTS ⚡️ Build success in 5008ms
@fedimint/ui:build: DTS dist/index.d.ts 7.13 KB
@fedimint/guardian-ui:dev: cache bypass, force executing 02229c4eafcd650b
@fedimint/guardian-ui:dev: $ react-scripts start
@fedimint/types:dev: $ tsup --watch
@fedimint/utils:dev: $ yarn run tsup --watch
@fedimint/ui:dev: $ tsup --watch
@fedimint/types:dev: CLI Building entry: src/index.ts
@fedimint/types:dev: CLI Using tsconfig: tsconfig.json
@fedimint/types:dev: CLI tsup v5.12.9
@fedimint/types:dev: CLI Using tsup config: /home/ethan/code/ui/packages/types/tsup.config.json
@fedimint/types:dev: CLI Running in watch mode
@fedimint/types:dev: CLI Target: node12
@fedimint/types:dev: CJS Build start
@fedimint/ui:dev: CLI Building entry: src/index.tsx
@fedimint/ui:dev: CLI Using tsconfig: tsconfig.json
@fedimint/ui:dev: CLI tsup v5.12.9
@fedimint/ui:dev: CLI Using tsup config: /home/ethan/code/ui/packages/ui/tsup.config.json
@fedimint/ui:dev: CLI Running in watch mode
@fedimint/ui:dev: CLI Target: node12
@fedimint/ui:dev: CJS Build start
@fedimint/types:dev: CJS ⚡️ Build success in 61ms
@fedimint/types:dev: CJS dist/index.js     2.54 KB
@fedimint/types:dev: CJS dist/index.js.map 6.30 KB
@fedimint/types:dev: CLI Watching for changes in "."
@fedimint/types:dev: CLI Ignoring changes in "**/{.git,node_modules}/**" | "dist"
@fedimint/utils:dev: $ /home/ethan/code/ui/node_modules/.bin/tsup --watch
@fedimint/ui:dev: CJS ⚡️ Build success in 74ms
@fedimint/ui:dev: CJS dist/index.js     39.12 KB
@fedimint/ui:dev: CJS dist/index.js.map 55.42 KB
@fedimint/ui:dev: CLI Watching for changes in "."
@fedimint/ui:dev: CLI Ignoring changes in "**/{.git,node_modules}/**" | "dist"
@fedimint/ui:dev: node:internal/errors:496
@fedimint/ui:dev:     ErrorCaptureStackTrace(err);
@fedimint/ui:dev:     ^
@fedimint/ui:dev: 
@fedimint/ui:dev: Error: EMFILE: too many open files, watch '.'
@fedimint/ui:dev:     at FSWatcher.<computed> (node:internal/fs/watchers:247:19)
@fedimint/ui:dev:     at Object.watch (node:fs:2418:34)
@fedimint/ui:dev:     at createFsWatchInstance (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js:119:1
5)
@fedimint/ui:dev:     at setFsWatchListener (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js:166:15)
@fedimint/ui:dev:     at NodeFsHandler._watchWithNodeFs (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler
.js:331:14)
@fedimint/ui:dev:     at NodeFsHandler._handleDir (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js:56
7:19)
@fedimint/ui:dev:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
@fedimint/ui:dev:     at async NodeFsHandler._addToNodeFs (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handl
er.js:617:16)
@fedimint/ui:dev:     at async /home/ethan/code/ui/node_modules/chokidar/index.js:451:21
@fedimint/ui:dev:     at async Promise.all (index 0)
@fedimint/ui:dev: Emitted 'error' event on FSWatcher instance at:
@fedimint/ui:dev:     at FSWatcher._handleError (/home/ethan/code/ui/node_modules/chokidar/index.js:647:10)
@fedimint/ui:dev:     at NodeFsHandler._addToNodeFs (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js:
645:18)
@fedimint/ui:dev:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
@fedimint/ui:dev:     at async /home/ethan/code/ui/node_modules/chokidar/index.js:451:21
@fedimint/ui:dev:     at async Promise.all (index 0) {
@fedimint/ui:dev:   errno: -24,
@fedimint/ui:dev:   syscall: 'watch',
@fedimint/ui:dev:   code: 'EMFILE',
@fedimint/ui:dev:   path: '.',
@fedimint/ui:dev:   filename: '.'
@fedimint/ui:dev: }
@fedimint/ui:dev: 
@fedimint/ui:dev: Node.js v18.18.2
@fedimint/ui:dev: error Command failed with exit code 1.
@fedimint/ui:dev: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@fedimint/utils:dev: CLI Building entry: src/index.tsx
@fedimint/utils:dev: CLI Using tsconfig: tsconfig.json
@fedimint/utils:dev: CLI tsup v5.12.9
@fedimint/utils:dev: CLI Using tsup config: /home/ethan/code/ui/packages/utils/tsup.config.json
@fedimint/utils:dev: CLI Running in watch mode
@fedimint/utils:dev: CLI Target: node12
@fedimint/utils:dev: CJS Build start
@fedimint/utils:dev: CJS ⚡️ Build success in 30ms
@fedimint/utils:dev: CJS dist/index.js     2.64 KB
@fedimint/utils:dev: CJS dist/index.js.map 2.36 KB
@fedimint/utils:dev: CLI Watching for changes in "."
@fedimint/utils:dev: CLI Ignoring changes in "**/{.git,node_modules}/**" | "dist"
@fedimint/utils:dev: node:internal/errors:496
@fedimint/utils:dev:     ErrorCaptureStackTrace(err);
@fedimint/utils:dev:     ^
@fedimint/utils:dev: 
@fedimint/utils:dev: Error: EMFILE: too many open files, watch '.'
@fedimint/utils:dev:     at FSWatcher.<computed> (node:internal/fs/watchers:247:19)
@fedimint/utils:dev:     at Object.watch (node:fs:2418:34)
@fedimint/utils:dev:     at createFsWatchInstance (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js:11
9:15)
@fedimint/utils:dev:     at setFsWatchListener (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js:166:1
5)
@fedimint/utils:dev:     at NodeFsHandler._watchWithNodeFs (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-hand
ler.js:331:14)
@fedimint/utils:dev:     at NodeFsHandler._handleDir (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.js
:567:19)
@fedimint/utils:dev:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
@fedimint/utils:dev:     at async NodeFsHandler._addToNodeFs (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-ha
ndler.js:617:16)
@fedimint/utils:dev:     at async /home/ethan/code/ui/node_modules/chokidar/index.js:451:21
@fedimint/utils:dev:     at async Promise.all (index 0)
@fedimint/utils:dev: Emitted 'error' event on FSWatcher instance at:
@fedimint/utils:dev:     at FSWatcher._handleError (/home/ethan/code/ui/node_modules/chokidar/index.js:647:10)
@fedimint/utils:dev:     at NodeFsHandler._addToNodeFs (/home/ethan/code/ui/node_modules/chokidar/lib/nodefs-handler.
js:645:18)
@fedimint/utils:dev:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
@fedimint/utils:dev:     at async /home/ethan/code/ui/node_modules/chokidar/index.js:451:21
@fedimint/utils:dev:     at async Promise.all (index 0) {
@fedimint/utils:dev:   errno: -24,
@fedimint/utils:dev:   syscall: 'watch',
@fedimint/utils:dev:   code: 'EMFILE',
@fedimint/utils:dev:   path: '.',
@fedimint/utils:dev:   filename: '.'
@fedimint/utils:dev: }
@fedimint/utils:dev: 
@fedimint/utils:dev: Node.js v18.18.2
@fedimint/utils:dev: error Command failed with exit code 1.
@fedimint/utils:dev: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@fedimint/utils:dev: error Command failed with exit code 1.
@fedimint/utils:dev: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@fedimint/types:dev: DTS Build start
@fedimint/ui:dev: ERROR: command finished with error: command (/home/ethan/code/ui/packages/ui) yarn run dev exited (
1)
command (/home/ethan/code/ui/packages/ui) yarn run dev exited (1)

 Tasks:    3 successful, 7 total
Cached:    3 cached, 7 total
  Time:    4.723s 

 ERROR  run failed: command  exited (1)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
┌─[ethan][winterberrylane][±][config-json-reboot ✓][~/code/ui]
└─▪ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 126742
max locked memory           (kbytes, -l) 4064892
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 126742
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
EthnTuttle commented 7 months ago

If I nuke the repo and then re-clone and run. the dev env seems to work fine. After several restarts, I begin to get issues

maan2003 commented 7 months ago

feels like node file watcher, maybe you need to ignore some directory?

wbobeirne commented 7 months ago

Give turbo an upgrade and see if that helps, we're quite a few versions behind and there may be some improvements around file watching.

EthnTuttle commented 7 months ago

feels like node file watcher, maybe you need to ignore some directory? This makes sense but I'm not sure what would be acceptable to ignore. When we run just mprocs there are at least five node instances that spin up in dev mode to watch guardian ui file and gateway files. With runtime config, we should only need one node watcher and then redirect where the config comes from using some middleware.

I will explore how we might be able to reduce dev env down to one node watcher per app, this might alleviate the problem and should be a general improvement to dev setup.

Give turbo an upgrade and see if that helps, we're quite a few versions behind and there may be some improvements around file watching.

yarn upgrade turbo && yarn clean bumped me from 1.9 to 1.12 for Turbo. Still get two instances of node failing with the same error:

yarn run v1.22.19
$ turbo run dev --filter=guardian-ui...
• Packages in scope: @fedimint/eslint-config, @fedimint/guardian-ui, @fedimint/tsconfig, @fedimint/types, @fedimint/u
i, @fedimint/utils
• Running dev in 6 packages
• Remote caching disabled
@fedimint/types:dev: cache bypass, force executing 9ec5c1f49145f9e5
@fedimint/types:dev: 
$ tsup --watch
@fedimint/types:build: cache hit, replaying logs 0df76758b9eec9b7
@fedimint/types:build: 
$ tsup
@fedimint/types:build: CLI Building entry: src/index.ts
@fedimint/types:build: CLI Using tsconfig: tsconfig.json
@fedimint/types:build: CLI tsup v5.12.9
@fedimint/types:build: CLI Using tsup config: /home/ethan/code/ui/packages/types/tsup.config.json
@fedimint/types:build: CLI Target: node12
@fedimint/types:build: CJS Build start
@fedimint/types:build: CJS ⚡️ Build success in 26ms
@fedimint/types:build: CJS dist/index.js     2.54 KB
@fedimint/types:build: CJS dist/index.js.map 6.30 KB
@fedimint/types:build: DTS Build start
@fedimint/types:build: DTS ⚡️ Build success in 1967ms
@fedimint/types:build: DTS dist/index.d.ts 6.15 KB
@fedimint/utils:dev: cache bypass, force executing 44abacc5b467fd09
@fedimint/utils:dev: 
@fedimint/types:dev: CLI Building entry: src/index.ts
@fedimint/types:dev: CLI Using tsconfig: tsconfig.json
@fedimint/types:dev: CLI tsup v5.12.9
@fedimint/types:dev: CLI Using tsup config: /home/ethan/code/ui/packages/types/tsup.config.json
@fedimint/types:dev: CLI Running in watch mode
@fedimint/types:dev: CLI Target: node12
@fedimint/types:dev: CJS Build start
@fedimint/types:dev: CJS ⚡️ Build success in 23ms
@fedimint/types:dev: CJS dist/index.js     2.54 KB
@fedimint/types:dev: CJS dist/index.js.map 6.30 KB
@fedimint/types:dev: CLI Watching for changes in "."
@fedimint/types:dev: CLI Ignoring changes in "**/{.git,node_modules}/**" | "dist"
$ yarn run tsup --watch
@fedimint/utils:build: cache hit, replaying logs 4502352973269397
@fedimint/utils:build: 
$ yarn run tsup
$ /home/ethan/code/ui/node_modules/.bin/tsup
@fedimint/utils:build: CLI Building entry: src/index.tsx
@fedimint/utils:build: CLI Using tsconfig: tsconfig.json
@fedimint/utils:build: CLI tsup v5.12.9
@fedimint/utils:build: CLI Using tsup config: /home/ethan/code/ui/packages/utils/tsup.config.json
@fedimint/utils:build: CLI Target: node12
@fedimint/utils:build: CJS Build start
@fedimint/utils:build: CJS ⚡️ Build success in 25ms
@fedimint/utils:build: CJS dist/index.js     2.64 KB
@fedimint/utils:build: CJS dist/index.js.map 2.36 KB
@fedimint/utils:build: DTS Build start
@fedimint/utils:build: DTS ⚡️ Build success in 1840ms
@fedimint/utils:build: DTS dist/index.d.ts 751.00 B
@fedimint/ui:dev: cache bypass, force executing 5545d2ea8fced084
@fedimint/ui:dev: 
@fedimint/ui:build: cache hit, replaying logs 96ee994e6a128fee
@fedimint/ui:build: 
$ tsup
@fedimint/ui:build: CLI Building entry: src/index.tsx
@fedimint/ui:build: CLI Using tsconfig: tsconfig.json
@fedimint/ui:build: CLI tsup v5.12.9
@fedimint/ui:build: CLI Using tsup config: /home/ethan/code/ui/packages/ui/tsup.config.json
@fedimint/ui:build: CLI Target: node12
@fedimint/ui:build: CJS Build start
@fedimint/ui:build: CJS ⚡️ Build success in 21ms
@fedimint/ui:build: CJS dist/index.js     39.36 KB
@fedimint/ui:build: CJS dist/index.js.map 55.72 KB
@fedimint/ui:build: DTS Build start
@fedimint/ui:build: DTS ⚡️ Build success in 4811ms
@fedimint/ui:build: DTS dist/index.d.ts 7.31 KB
@fedimint/guardian-ui:dev: cache bypass, force executing 53c4fd867fbdbb53
@fedimint/guardian-ui:dev: 
@fedimint/types:dev: DTS Build start
$ /home/ethan/code/ui/node_modules/.bin/tsup --watch
$ tsup --watch
@fedimint/utils:dev: CLI Building entry: src/index.tsx
@fedimint/utils:dev: CLI Using tsconfig: tsconfig.json
@fedimint/utils:dev: CLI tsup v5.12.9
@fedimint/utils:dev: CLI Using tsup config: /home/ethan/code/ui/packages/utils/tsup.config.json
@fedimint/utils:dev: CLI Running in watch mode
@fedimint/ui:dev: CLI Building entry: src/index.tsx
Starting the development server...
@fedimint/guardian-ui:dev: 
@fedimint/types:dev: DTS ⚡️ Build success in 3695ms
@fedimint/utils:dev: 
@fedimint/utils:dev: node:internal/event_target:1033
@fedimint/utils:dev:   process.nextTick(() => { throw err; });
@fedimint/utils:dev:                            ^
@fedimint/utils:dev: Error: EMFILE: too many open files, watch '/home/ethan/code/ui/packages/utils/src/index.tsx'
@fedimint/utils:dev:     at FSWatcher.<computed> (node:internal/fs/watchers:247:19)
@fedimint/utils:dev:     at Object.watch (node:fs:2418:34)
@fedimint/utils:dev:     at createFsWatchInstance (/home/ethan/code/ui/node_modules/rollup/dist/shared/index.js:2541:
17)
@fedimint/utils:dev:     at setFsWatchListener (/home/ethan/code/ui/node_modules/rollup/dist/shared/index.js:2588:15)
@fedimint/utils:dev:     at NodeFsHandler$1._watchWithNodeFs (/home/ethan/code/ui/node_modules/rollup/dist/shared/ind
ex.js:2743:14)
@fedimint/utils:dev:     at NodeFsHandler$1._handleFile (/home/ethan/code/ui/node_modules/rollup/dist/shared/index.js
:2807:23)
@fedimint/utils:dev:     at NodeFsHandler$1._addToNodeFs (/home/ethan/code/ui/node_modules/rollup/dist/shared/index.j
s:3049:21)
@fedimint/utils:dev:     at async /home/ethan/code/ui/node_modules/rollup/dist/shared/index.js:4043:21
@fedimint/utils:dev:     at async Promise.all (index 0)
@fedimint/utils:dev: Emitted 'error' event on Worker instance at:
@fedimint/utils:dev:     at [kOnErrorMessage] (node:internal/worker:300:10)
@fedimint/utils:dev:     at [kOnMessage] (node:internal/worker:311:37)
@fedimint/utils:dev:     at MessagePort.<anonymous> (node:internal/worker:212:57)
@fedimint/utils:dev:     at [nodejs.internal.kHybridDispatch] (node:internal/event_target:757:20)
@fedimint/utils:dev:     at exports.emitMessage (node:internal/per_context/messageport:23:28) {
@fedimint/utils:dev:   errno: -24,
@fedimint/utils:dev:   syscall: 'watch',
@fedimint/utils:dev:   code: 'EMFILE',
@fedimint/utils:dev:   path: '/home/ethan/code/ui/packages/utils/src/index.tsx',
@fedimint/utils:dev:   filename: '/home/ethan/code/ui/packages/utils/src/index.tsx'
@fedimint/utils:dev: }
@fedimint/utils:dev: 
@fedimint/utils:dev: Node.js v18.18.2
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@fedimint/utils:dev: ERROR: command finished with error: command (/home/ethan/code/ui/packages/utils) /tmp/nix-shell.
YfOJ33/yarn--1707390570027-0.7353336183480703/yarn run dev exited (1)
@fedimint/utils#dev: command (/home/ethan/code/ui/packages/utils) /tmp/nix-shell.YfOJ33/yarn--1707390570027-0.7353336
183480703/yarn run dev exited (1)

 Tasks:    3 successful, 7 total
Cached:    3 cached, 7 total
  Time:    5.667s 
Failed:    @fedimint/utils#dev

 ERROR  run failed: command  exited (1)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
elsirion commented 7 months ago

In Fedimint we had similar problems due to socket connections in tests iirc (everything is a file in UNIX after all …) and solved it by making the user set the limit higher. Might be worth playing around with ulimit.

                  if [ "$(ulimit -Sn)" -lt "1024" ]; then
                      >&2 echo "⚠️  ulimit too small. Run 'ulimit -Sn 1024' to avoid problems running tests"
                  fi

https://github.com/fedimint/fedimint/blob/47eb296afc274c9526163993141edc42f8c8c9d1/flake.nix#L323-L325

EthnTuttle commented 7 months ago

In Fedimint we had similar problems due to socket connections in tests iirc (everything is a file in UNIX after all …) and solved it by making the user set the limit higher. Might be worth playing around with ulimit.

                  if [ "$(ulimit -Sn)" -lt "1024" ]; then
                      >&2 echo "⚠️  ulimit too small. Run 'ulimit -Sn 1024' to avoid problems running tests"
                  fi

https://github.com/fedimint/fedimint/blob/47eb296afc274c9526163993141edc42f8c8c9d1/flake.nix#L323-L325

I came across this in my research but ulimit -n is already 1024 so I'm not sure that is the issue.

EthnTuttle commented 7 months ago

On my desktop in the office, upgrading turbo seemingly has resolved the issue, I was able to restart mprocs many times an none of the builds failed. Both running Ubuntu 22.04.

elsirion commented 6 months ago

I came across this in my research but ulimit -n is already 1024 so I'm not sure that is the issue.

  1. ulimit supports more options, I don't know all of them/what might be relevant off the top of my head though
  2. Maybe 1024 is still too low in this case?

But good if it works now.