Sequencing Server crashes creating a large expansion set

[cartermak]

Checked for duplicates

Yes - I've already checked

Is this a regression?

No - This is a new bug

Version

1.6.1

Describe the bug

When creating a moderate-sized expansion set (73 activity types) on a local instance of Aerie, the sequencing server crashes and fails to create the expansion set (presumably, during the process of validating the expansion rules against the activity schema). Increasing the memory allocated to Docker to 32 GB seems to have resolved the issue, but I haven't tested this extensively.

Reproduction

On a local instance of Aerie with Docker allocated 16 GB of memory, create an expansion set with >70 activity types (arbitrary number). The container crashes and restarts; Hasura returns a "http exception when calling webhook" for the createExpansionSet action.

A possible compounding factor is the size of the command dictionary.

Logs included below from the sequencing server container, although they don't seem too insightful.

Logs

0 verbose cli /usr/local/bin/node /usr/local/bin/npm
1 info using npm@8.19.3
2 info using node@v18.13.0
3 timing npm:load:whichnode Completed in 0ms
4 timing config:load:defaults Completed in 0ms
5 timing config:load:file:/usr/local/lib/node_modules/npm/npmrc Completed in 0ms
6 timing config:load:builtin Completed in 1ms
7 timing config:load:cli Completed in 1ms
8 timing config:load:env Completed in 1ms
9 timing config:load:file:/app/.npmrc Completed in 2ms
10 timing config:load:project Completed in 7ms
11 timing config:load:file:/root/.npmrc Completed in 0ms
12 timing config:load:user Completed in 0ms
13 timing config:load:file:/usr/local/etc/npmrc Completed in 0ms
14 timing config:load:global Completed in 0ms
15 timing config:load:validate Completed in 1ms
16 timing config:load:credentials Completed in 1ms
17 timing config:load:setEnvs Completed in 0ms
18 timing config:load Completed in 13ms
19 timing npm:load:configload Completed in 14ms
20 timing npm:load:mkdirpcache Completed in 1ms
21 timing npm:load:mkdirplogs Completed in 1ms
22 verbose title npm start
23 verbose argv "start"
24 timing npm:load:setTitle Completed in 1ms
25 timing config:load:flatten Completed in 2ms
26 timing npm:load:display Completed in 2ms
27 verbose logfile logs-max:10 dir:/root/.npm/_logs
28 verbose logfile /root/.npm/_logs/2023-07-11T18_51_17_281Z-debug-0.log
29 timing npm:load:logFile Completed in 4ms
30 timing npm:load:timers Completed in 0ms
31 timing npm:load:configScope Completed in 0ms
32 timing npm:load Completed in 23ms
33 silly logfile done cleaning log files
34 timing command:run-script Completed in 195059ms
35 timing command:start Completed in 195063ms
36 verbose stack Error: command failed
36 verbose stack     at ChildProcess.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/@npmcli/promise-spawn/lib/index.js:63:27)
36 verbose stack     at ChildProcess.emit (node:events:513:28)
36 verbose stack     at maybeClose (node:internal/child_process:1091:16)
36 verbose stack     at ChildProcess._handle.onexit (node:internal/child_process:302:5)
37 verbose pkgid sequencing-server@1.6.1
38 verbose cwd /app
39 verbose Linux 5.10.104-linuxkit
40 verbose node v18.13.0
41 verbose npm  v8.19.3
42 error path /app
43 error command failed
44 error signal SIGKILL
45 error command sh -c -- node --experimental-vm-modules ./build/app.js
46 verbose exit 1
47 timing npm Completed in 195095ms
48 verbose code 1
49 error A complete log of this run can be found in:
49 error     /root/.npm/_logs/2023-07-11T18_51_17_281Z-debug-0.log

System Info

Docker Desktop v4.11.1 on MacOS 13.4.1

Workaround

Create expansion sets with fewer activities

Severity

Moderate

[dandelany]

First half of this is in PR https://github.com/NASA-AMMOS/aerie/pull/1324

[dandelany]

Second half PR: #1332

[joswig]

@goetzrrGit does #1475 fix close this?