Open coryjreid opened 1 year ago
Interesting, it looks like tsx may be misbehaving. Could you check out the built image in the PR above and see if that fixes it?
If tsx
is the root issue, maybe I'll switch back to pre-transpiling for faster startup times as well.
It appears the issue still exists, albeit the rate does seem slower.
I cloned the repo, checked out the branch, and ran docker build -t test .
from the repository root. Then I modified my compose file to use the specific image that was built. If this fails to properly test please do let me know (I'm not super savvy on Docker).
Below you can see relevant command outputs.
root@muse:~# du -cs /var/lib/docker/*
3188 /var/lib/docker/buildkit
40 /var/lib/docker/containers
4 /var/lib/docker/engine-id
2004 /var/lib/docker/image
52 /var/lib/docker/network
3217924 /var/lib/docker/overlay2
16 /var/lib/docker/plugins
4 /var/lib/docker/runtimes
4 /var/lib/docker/swarm
4 /var/lib/docker/tmp
28 /var/lib/docker/volumes
3223268 total
root@muse:~# du -cs /var/lib/docker/*
3188 /var/lib/docker/buildkit
40 /var/lib/docker/containers
4 /var/lib/docker/engine-id
2004 /var/lib/docker/image
52 /var/lib/docker/network
3218516 /var/lib/docker/overlay2
16 /var/lib/docker/plugins
4 /var/lib/docker/runtimes
4 /var/lib/docker/swarm
4 /var/lib/docker/tmp
28 /var/lib/docker/volumes
3223860 total
root@muse:~# docker diff muse
A /data
C /tmp
A /tmp/yarn--1693530312744-0.025729490385082876
A /tmp/yarn--1693530312744-0.025729490385082876/node
A /tmp/yarn--1693530312744-0.025729490385082876/yarn
A /tmp/capacitor-8424f91a84d4c944638c8e3e9e185a58.tmp
A /tmp/envset-database-url-24d42b41.sh
C /root
C /root/.cache
C /root/.cache/checkpoint-nodejs
A /root/.cache/checkpoint-nodejs/prisma-82996626
A /root/.cache/checkpoint-nodejs/signature
C /root/.npm
A /root/.npm/_update-notifier-last-checked
A /root/.npm/_cacache
A /root/.npm/_cacache/content-v2
A /root/.npm/_cacache/content-v2/sha512
A /root/.npm/_cacache/content-v2/sha512/eb
A /root/.npm/_cacache/content-v2/sha512/eb/8d
A /root/.npm/_cacache/content-v2/sha512/eb/8d/75c8d2d874c09ea42b0dc0cca972c460d74f1913183dc199c436110fcdd69bb03cb992e5c99140c6f1628f7bcbdaf6cc3412985de0294cd9d 9c8fc5749c6
A /root/.npm/_cacache/index-v5
A /root/.npm/_cacache/index-v5/46
A /root/.npm/_cacache/index-v5/46/e0
A /root/.npm/_cacache/index-v5/46/e0/f061ea30b750e78f6672057dcdbe3e1e78ac7b48c4593ab70b28b49f02bd
A /root/.npm/_cacache/tmp
C /root/.npm/_logs
A /root/.npm/_logs/2023-09-01T01_05_13_158Z-debug-0.log
root@muse:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
test latest d25ee43030e2 8 minutes ago 1.13GB
root@muse:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bbf601cc6ca7 d25ee43030e2 "docker-entrypoint.s…" 7 minutes ago Up 7 minutes muse
The comment here has a image tag that you can pull directly to get the PR code instead of building it yourself. But yes, the new output looks correct (no more esbuild caching).
I'll try to reproduce this locally.
I left Muse running for about 10m. I observed overlay2
increasing by about 100MB at the begining but otherwise didn't grow, so having a hard time reproducing this.
When you're observing the behavior, could you go into the container with docker exec -it <container-id> /bin/bash
, run du -h /tmp
, and post the output?
So we've been playing music with the bot for several hours a day every day and the overlay2
file has now grown to over 8GB. While streaming music the behavior continues. Output sample:
root@muse:~# du -cs /var/lib/docker/*
3188 /var/lib/docker/buildkit
40 /var/lib/docker/containers
4 /var/lib/docker/engine-id
2004 /var/lib/docker/image
52 /var/lib/docker/network
8686732 /var/lib/docker/overlay2
16 /var/lib/docker/plugins
4 /var/lib/docker/runtimes
4 /var/lib/docker/swarm
4 /var/lib/docker/tmp
28 /var/lib/docker/volumes
8692076 total
root@muse:~# du -cs /var/lib/docker/*
3188 /var/lib/docker/buildkit
40 /var/lib/docker/containers
4 /var/lib/docker/engine-id
2004 /var/lib/docker/image
52 /var/lib/docker/network
8688028 /var/lib/docker/overlay2
16 /var/lib/docker/plugins
4 /var/lib/docker/runtimes
4 /var/lib/docker/swarm
4 /var/lib/docker/tmp
28 /var/lib/docker/volumes
8693372 total
The output you requested is as follows:
root@muse:/tmp# du -h /tmp
12K /tmp/yarn--1693530312744-0.025729490385082876
12K /tmp/yarn--1693530219317-0.01351705037614681
2.2M /tmp/v8-compile-cache-0/10.2.154.13-node.9
2.2M /tmp/v8-compile-cache-0
2.7G /tmp
Running without -h
:
root@muse:/tmp# du /tmp
12 /tmp/yarn--1693530312744-0.025729490385082876
12 /tmp/yarn--1693530219317-0.01351705037614681
2160 /tmp/v8-compile-cache-0/10.2.154.13-node.9
2164 /tmp/v8-compile-cache-0
2749288 /tmp
root@muse:/tmp# du /tmp
12 /tmp/yarn--1693530312744-0.025729490385082876
12 /tmp/yarn--1693530219317-0.01351705037614681
2160 /tmp/v8-compile-cache-0/10.2.154.13-node.9
2164 /tmp/v8-compile-cache-0
2749364 /tmp
I notice /tmp
is still growing. I'm going to investigate further and will post another update.
I ran a couple du /tmp/*
commands sequentially (while the bot is actively streaming) and saved the output. Then I later diff
ed the output.
root@muse:/tmp# du /tmp/* > output1
root@muse:/tmp# du /tmp/* > output2
root@muse:/tmp# diff output1 output2
39c39
< 10764 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp
---
> 10860 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp
44a45
> 4 /tmp/output1
And the command which launched the process which is generating the data:
/usr/local/bin/node --require /usr/app/node_modules/tsx/dist/suppress-warnings.cjs --loader file:///usr/app/node_modules/tsx/dist/loader.js src/scripts/migrate-and-start.ts
I captured this by installing procps
and running ps -p <pid> -o command
.
root@muse:/tmp# du -h /tmp 12K /tmp/yarn--1693530312744-0.025729490385082876 12K /tmp/yarn--1693530219317-0.01351705037614681 2.2M /tmp/v8-compile-cache-0/10.2.154.13-node.9 2.2M /tmp/v8-compile-cache-0 2.7G /tmp
This is strange, du
reports /tmp
is 2.7 GB but there's no file/directory that big? 🤔
root@muse:/tmp# du /tmp/* > output1 root@muse:/tmp# du /tmp/* > output2 root@muse:/tmp# diff output1 output2 39c39 < 10764 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp --- > 10860 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp 44a45 > 4 /tmp/output1
Was this when you were observing the 2.7 GB /tmp
directory or had you restarted Muse between runs? In theory the temp file for the capacitor package shouldn't infinitely grow, but here it's only 11 MB.
Problematic Behavior
Something within the Docker container appears to be writing to the container's filesystem causing it to grow infinitely in size, resulting in the container's
overlay2
file to eventually fill the disk.Expected Behavior
Container doesn't write to un-mounted directories within the container (or, at least not continually).
Background
This VM only hosts this Docker container and an NGINX server which is a forward-proxy for some streaming URLs that need to remain private and has 32GB of disk space allocated to it.
In the below output you can see me run
du -cs /var/lib/docker/*
back-to-back with a period of only a few seconds between the two and the size of theoverlay2
folder growing in that short time (the bot is actively streaming in Discord). You can also see I only have the one Docker container running.This StackOverflow post suggested looking into files being written in the container by running
docker container diff <container_name/id>
, which I did, and you can see the output here.For the sake of completeness, my compose file is: