[bug] app crashes due to out of memory error

[cannawen]

• [ ] Fix crash
• [ ] Fix memory leak warning
• [ ] can we stream data directly to axios?

When there are multiple people talking on discord, the bot crashes because it cannot handle so much data

2023-04-30T23:32:45.487 app[e2865644f6e686] yyz [info] [ 6111.388520] Out of memory: Killed process 521 (node) total-vm:32835100kB, anon-rss:182896kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2084kB oom_score_adj:0

Crash at 7:32pm -

[cannawen]

The crash happened before a game started; there were 6 of us in Discord and maybe 2-3 people talking at once. My hypothesis is that the app was trying to juggle too much uncompressed audio data at the same time, causing it to run out of memory.

Refactored code to stream the audio to disk (instead of keeping it as a buffer in memory) and then when it is done, read the data from disk to send off to the network to do the STT.

[cannawen]

Looks like it held up pretty well with 6 people talking after the change. No crashes.

We are getting a bunch of new errors that look kinda sus though:

2023-05-01T02:21:55.293 app[e2865644f6e686] yyz [info] (node:521) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 end listeners added to [AudioReceiveStream]. Use emitter.setMaxListeners() to increase limit

2023-05-01T02:21:55.661 app[e2865644f6e686] yyz [info] (node:521) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [AudioReceiveStream]. Use emitter.setMaxListeners() to increase limit

2023-05-01T02:21:55.661 app[e2865644f6e686] yyz [info] (node:521) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [AudioReceiveStream]. Use emitter.setMaxListeners() to increase limit

2023-05-01T02:30:29.830 app[e2865644f6e686] yyz [info] (node:521) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 end listeners added to [AudioReceiveStream]. Use emitter.setMaxListeners() to increase limit

[cannawen]

Pipeline does not automatically close destination stream.

https://nodejs.org/api/stream.html#streampipelinesource-transforms-destination-options

[cannawen]

TODO - look into if we can stream data directly to network

[cannawen]

Adding .close on the out stream does not fix the memory leak warning

[cannawen]

Closing this issue because it's no longer crashing; opened #71 and #72 for open tasks

[cannawen]

Re-opening. Crashed again.

2023-05-06T22:43:53.370 app[e2865644f6e686] yyz [info] [55924.382968] Out of memory: Killed process 521 (node) total-vm:32836244kB, anon-rss:184884kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2080kB oom_score_adj:0

2023-05-06T22:43:53.379 proxy[e2865644f6e686] yyz [error] could not make HTTP request to instance: connection error: Connection reset by peer (os error 104)

2023-05-06T22:43:53.379 proxy[e2865644f6e686] yyz [error] could not make HTTP request to instance: connection error: Connection reset by peer (os error 104)

2023-05-06T22:43:53.379 proxy[e2865644f6e686] yyz [error] could not make HTTP request to instance: connection error: Connection reset by peer (os error 104)

2023-05-06T22:43:53.560 app[e2865644f6e686] yyz [info] Starting clean up.

2023-05-06T22:43:53.560 app[e2865644f6e686] yyz [info] Umounting /dev/vdb from /data

2023-05-06T22:43:53.562 app[e2865644f6e686] yyz [info] Process appears to have been OOM killed!

2023-05-06T22:43:54.028 proxy[e2865644f6e686] yyz [error] instance refused connection. is your app listening on 0.0.0.0:8080? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)

2023-05-06T22:43:54.562 app[e2865644f6e686] yyz [info] [55925.575762] reboot: Restarting system

2023-05-06T22:43:54.673 app[e2865644f6e686] yyz [info] Out of memory: Killed process

2023-05-06T22:43:55.809 app[e2865644f6e686] yyz [info] Starting init (commit: 083df78)...

2023-05-06T22:43:55.855 app[e2865644f6e686] yyz [info] Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755

2023-05-06T22:43:55.857 app[e2865644f6e686] yyz [info] Preparing to run: `node ./build/index.js` as root

2023-05-06T22:43:55.870 app[e2865644f6e686] yyz [info] 2023/05/06 22:43:55 listening on [fdaa:1:c218:a7b:f0:1117:50b7:2]:22 (DNS: [fdaa::3]:53)

2023-05-06T22:43:57.027 app[e2865644f6e686] yyz [info] 2023-05-06 22:43:57 info [APP] Starting server on http://0.0.0.0:8080

2023-05-06T22:43:58.312 proxy[e2865644f6e686] yyz [warn] Failed to proxy HTTP request (error: no known healthy instances found for route tcp/80. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 10)

Failed to proxy HTTP request (error: no known healthy instances found for route tcp/80. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 20 ms (attempt 1)

2023-05-06T22:44:02.845 app[e2865644f6e686] yyz [info] 2023-05-06 22:44:02 info [DISCORD] Discord ready with bot: dota-coach#9402

Failed to proxy HTTP request (error: no known healthy instances found for route tcp/80. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the 'immediate' strategy? if not, this could be a delayed state issue)). Retrying in 1000 ms (attempt 10)

[cannawen]

There were 5 of us in the chat, possibly talking at the same time

[cannawen]

Hypothesis - long-running audio eating up all the memory (i.e. music bot, someone's noisy mic or very long-winded rant) Solution - after 5 seconds, stop trying to transcribe what a user is saying

Kind of weird this is still happening when we stream to file though

[cannawen]

Closing; it didn't crash yesterday, so...