Open chrishoage opened 1 year ago
Can you please provide the flexo version you're using? (e.g. docker image ls nroi/flexo
).
Also, please provide more details about how the clients are interacting with flexo, for example:
I'm asking the second question because I've noticed some regularity in those log statements:
[2023-08-04T06:22:11.124Z INFO flexo] Received request for path "core/os/x86_64/core.db".
The core.db
file is the first file requested by pacman when you run pacman -Syu
or pacman -Sy
or similar. If you grep for those requests in the log statements, you get this:
[2023-08-04T06:22:11.124Z INFO flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:27:15.828Z INFO flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:32:20.478Z INFO flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:37:25.131Z INFO flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:42:29.479Z INFO flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
[2023-08-04T06:47:33.904Z INFO flexo] Received request for path "core/os/x86_64/core.db". Range start: 0
If you compare the timestamps, they are all 5 minutes and about 4.5 seconds apart. Even though I cannot rule out that this is caused by a flexo bug, I just want to ask if you have a cron job or something before I start chasing ghosts :wink:
Can you please provide the flexo version you're using? (e.g. docker image ls nroi/flexo).
❯ docker image ls nroi/flexo
REPOSITORY TAG IMAGE ID CREATED SIZE
nroi/flexo latest 6609a4785656 5 days ago 112MB
nroi/flexo <none> 601652172d6e 10 months ago 112MB
Have you noticed any errors or something unexpected when you run pacman? Do you have any scripts, cronjobs or whatever that are running pacman, or otherwise interacting with flexo? Or are you certain that all requests sent to flexo are done because an actual human is running the pacman command?
I have a polybar script which runs checkupdates every 5 minutes to check to see if there are any pending updates.
The logs may be a red herring in this case. The eating a lot of CPU until it restarts happens maybe once a month or so?
It doesn't "go away" and even happens when the system that runs checkupdates
is asleep and not hitting the flexo mirror for updates. In my screenshot above for example in my screenshot above no computer on my network was asking for updates every 5 minutes.
So perhaps the logs are not showing what is eating the CPU.
I didn't have the presence of mind to inspect the container to see what process was eating CPU (I restarted the container from bed with my phone since I could hear my server was working hard for no reason :sweat_smile:) - next time this happens I will do so to provide additional information.
I didn't have the presence of mind to inspect the container to see what process was eating CPU
Alright. If it's not too much effort, and you have no concerns regarding privacy, please also provide a coredump (I don't expect any privacy-relevant data in the flexo coredump, but that's up to you). You can send it to me via e-mail: nroi@mailbox.org
Here's how to create a coredump on flexo running via docker:
docker exec -it <container-id> /bin/bash
kill -11 $(pidof flexo)
./var/lib/systemd/coredump
directory (if you're running ArchLinux).Also, if you have any logs available from the time when the CPU load was so high, that would also be helpful.
Unfortunately it looks like the logs rotated out and I'm having difficulty with coredump (running an ubuntu host at the moment)
This happens with some frequency so next time it does I will immediately gather logs and investigate how to generate a core dump when I have more time to dedicate to debugging this
In the mean time leaving this open may help gather information if anyone else has similar behavior (or not - which would indicate this is something broken with my setup)
Thank you for your help!
Describe the bug
Sometimes Flexo gets stuck in an update cycle that eats a ton of CPU. The logs below are abbreviated but go on for several hours.
The fall off in the image below is after i restarted the container.
I unfortunately have not figured out how to reproduce this. It hapens once every few weeks and has happened for many months now
Installation method Docker
Distribution default arch
Version If you've installed Flexo from AUR, provide the output of:
pacman -Qi flexo-git
. If you use Docker, provide the version tag of the Docker image you're using (checkdocker image ls nroi/flexo
for example).Log