High CPU, RAM and I/O usage

[SoWhy]

Describe the bug Okay, since a few days, even after the upgrade to the 2024 version, I noticed the VM on which Remotely is running will quickly start using a lot of CPU and RAM and I can literally watch the free space being eaten by the "overlay" device:

root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  2.2G   28G   8% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  180K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  2.2G   28G   8% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  6.1G   24G  21% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  6.1G   24G  21% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  7.1G   23G  24% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  7.1G   23G  24% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  8.0G   22G  27% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  8.0G   22G  27% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G  9.0G   21G  31% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G  9.0G   21G  31% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G   10G   20G  34% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G   10G   20G  34% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged
[...]
root@remotely-v2:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VMs-vm--901--disk--0   32G   18G   13G  59% /
none                              492K  4.0K  488K   1% /dev
udev                              126G     0  126G   0% /dev/tty
tmpfs                             126G     0  126G   0% /dev/shm
tmpfs                              51G  184K   51G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
overlay                            32G   18G   13G  59% /var/lib/docker/overlay2/cfba3a67d253b20ed27da9ee15744abf16395aa5963cac0866f02033252cc267/merged

For a while, it will only spike in intervals, filling up a lot of space before returning to 2.2G but after that, it will fill up everything and the CPU will keep being at 99% until I kill the server and restart which will only give me a short reprieve before the circle repeats htop will show (when it works) that "dotnet Remotely_Server.dll" is the culprit I assume that the problem is some runaway setting for logging but I cannot find it.

Trying to analyze the mount directory with ncdu shows no additional files, especially not of that size

I set up a new VM with a manual Debian install (instead of the Debian CT), installed Remotely fresh and copied the settings from the old install but the problem persists. It also persists when installing Debian and Docker on a fresh PC and copying over the configuration files from the old install

The problem does not seem to appear before accessing the web interface for the first time. So the service will run with 0% CPU for hours but once you access the web interface, it will start acting up.

Assigning more CPU cores will mitigate the problem but it's not a fix because it will still create spikes in usage, just not as high.

I found some information online that indicates that it might be related to dotnet 8.x but the problem started with my dotnet 7.x based version before the upgrade. Unfortunately, I cannot remember what changed on that day and I cannot find anything related in the log files.

To Reproduce Steps to reproduce the behavior:

1. Install fresh Debian and Remotely
2. Import Remotely.db etc. from old install
3. Access web interface

Remotely Version Server (can be found on about page): 2024.02.23.1927 Agent (can be found in device card): 2024.02.23.1927

Expected Behavior No runaway CPU, memory and I/O usage

Screenshots See https://www.reddit.com/r/remotely_app/comments/1ccyhs8/docker_overlay_eating_up_all_my_space_quickly_and/

Desktop (please complete the following information):

• OS: Windows 10/11 Pro
• Browser Firefox
• Version 126

Additional Context Add any other context about the problem here.

• [X] I am running Remotely in Docker and not on my QNAP, Synology, or Internet Connected Toaster

[bitbound]

What's in the logs on the server?

[SoWhy]

If you tell me which logs I should share, I will gladly do so

[bitbound]

If you tell me which logs I should share, I will gladly do so

Sorry. If you're using Docker, they will be in the mounted volume under the logs directory (relevant readme section).

If you're hosting "bare metal" with systemd, it will be in the content root directory (where the binaries are located), under /AppData/logs.

I'm basically wondering if there's an error that's spamming rapidly.

[SoWhy]

Sorry for the delay in replaying. The log files in /var/www/remotely/logs contain no information apart from what I did (so like six lines after a reboot). I ran ncdu on / while it was happening but it didn't find any additional files

[SoWhy]

If you're hosting "bare metal" with systemd, it will be in the content root directory (where the binaries are located), under /AppData/logs.

Is it still possible to run Remotely without docker, so I can test whether it's docker-related? if so, how?

[bitbound]

How many agents are connected to the server? Remotely won't scale well into the thousands or anything.

Is it still possible to run Remotely without docker

It's still possible. There's a ZIP with each release that contains linux-x64 binaries. You'd extract them, then set up a systemd service that executes the Remotely_Server binary. Then put a reverse proxy in front of it. I prefer caddy, since the default configuration has everything needed for Remotely to work correctly.

Below is an example systemd service and caddy configuration. Official docs can be found here.

/etc/systemd/system/remotely-server.service:

[Unit]
Description=Remotely Server

[Service]
WorkingDirectory=/var/www/remotely
ExecStart=/var/www/remotely/Remotely_Server --urls "http://*:5020"
Restart=always
RestartSec=10
SyslogIdentifier=remotely
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false

[Install]
WantedBy=multi-user.target

/etc/caddy/Caddyfile:

remotely.example.com {
  reverse_proxy 127.0.0.1:5020
}