Closed vudutech closed 1 year ago
Appreciate your bug report.
PC becomes unresponsive.
Ctrl+C
or open another terminal and kill the process or dump to a different virtual terminal? ssh
into the box from another machine and wait for this to take place, and try to kill the process?top
? Are you using lots of memory for instance? More than 200M would be extraordinary./usr/bin/X11
? Just for comparison, I have a Atom 4 core NAS with a SSD ZFS home directory and my %CPU in top
can spike to ~390%, if this was an uncached first run, but mostly keeps steady around 250%. I've never hung that system, and never hung a terminal during a deleted recursive run, but because you have so many more, faster cores, maybe you are just getting into trouble faster?
strace
like strace -f -o output.txt -- httm -d -R ~
?Please to assist you to diagnose. But if you want some help, would you also upgrade to 0.17.5
? Perhaps try the new PPA. Thanks. Maybe its been hammered out already? I was doing some work re: the recursive model at the time. Could be some weird interaction there?
You're welcome. Cool project.
Bit surprised myself at the number of sysmlinks!
Have attached strace results, head -n 100 and tail -n 100. Process again stalled after 1 or 2 mins on a PDF motherboard manual. Notably only 1 thread at 100% this time as shown in htop and system remained largely usable.
Also note that I had an issue with bluetooth not functioning today, coupled with or caused by, a power outage overnight which saw me zfs rollback (excluding the /home dir) 24 hours to regain a functioning system.
Will look to upgrade in the coming day or 2, and test again. I've really just been using your provided examples to get a feel for how httm works.
Hope this helps and thanks for your efforts. Let me know if I can assist further.
Some more strace detail. Original output.txt was >13GB
From what you're telling me, such high CPU usage is pretty normal. httm
will gladly use lots of threads, because deleted searches are IO and CPU intensive.
It's still amazing it me that it could stop such a powerful system in its tracks though, but I'm guessing that has something to do with it. I'd take three shots in the dark:
statx
calls (or whatever) and your drive gets a little backed up, and although your CPU is being "utilized", its actually stalled in iowait? httm
does all sync IO. Maybe async is required for such a fast machine?I'm betting its the confluence of all 3.
Some things to play with to get more info:
Check your %iowait
in another terminal window while running httm
with iostat -m 1 100
.
And play around with the # of threads you use to see what works best? You can set the # of threads used with and environment variable: export RAYON_NUM_THREADS=8
Sounds like a reasonable assessment.
Upgraded to the PPA version. Similar result. iostat and htop info attached.
iostat.txt
Somewhat better with "export RAYON_NUM_THREADS=8" but still sluggish. Htop shows cpu slightly lower and jumping around threads.
This PR, https://github.com/kimono-koans/httm/pull/59, should be a ~7x speedup for what I believe to be your issue. Here, we use snapshot birth times instead of file modify times for the policy of which deleted version to use for behind deleted dir enumeration. Should add to a release soon.
Probably not much more low lying performance fruit here.
I'm going to close unless you have something more particular. Again thanks for filing, and feel free to refile if the next release doesn't seem to fix and you have more data re: possible causes!
EDIT: Spoke too soon. https://github.com/kimono-koans/httm/commit/fc8419464f633826c357b5eda2f6cff4aa2abecb is ~60x faster on my system.
Glad I could help and thanks for your diligence.
Installed latest Update. Big Improvement!
httm -d -R ~ keeps chugging along, mostly. Some slowdowns when processing random files/dirs. Browser cache files and ".cache/plasma_theme_kubuntu_v20.04.3.kcache" seems to often be a problem,
All 32 threads seem to spike briefly then sit around 30-50%. Machine remains mostly responsive. Youtube paused at 1 point for around 10 seconds and then kept running.
CTRL C'd and restarted. Progressed further but occasionally stalls on random files, some only a few KiB's.
Little dots progress indicator stalls sometimes or is replaced with a solid block cursor. Walked away for a coffee (20mins) and the process completed.
Glad it works better.
Little dots progress indicator stalls sometimes or is replaced with a solid block cursor. Walked away for a coffee (20mins) and the process completed.
This is normal. If no new directories are being processed, then the progress indicator will stop. The indicator ticks as each directory is completed. When a directory isn't completed it doesn't tick. So, no tick could indicate a deadlock/stall, but in my experience it means, oops, this directory is super strange.
But I can't reproduce what is likely a pathological directory, unless I know more about what makes that directory/file pathological. I've tested directories with 500,000 files, created a few snapshots, then ran httm
, it completes within a few seconds. I regularly run on directories with 20,000 deleted files.
If it is pausing on a file/directory, what is special about that file or that directory? Like:
Browser cache files and ".cache/plasma_theme_kubuntu_v20.04.3.kcache" seems to often be a problem,
Two things:
What is the httm
output for ".cache/plasma_theme_kubuntu_v20.04.3.kcache" and other problematic files? Do those runs complete quickly? What about the containing directory? Are there a lot of snapshots of the file/containing directory, zfs list -t snapshot "$(httm -m -n '.cache/plasma_theme_kubuntu_v20.04.3.kcache')"
? How many other files are in the directory? How often do files get deleted from the directory? It's a cache so I'd imagine they get deleted all the time.
If you think httm
is slow, it may very well be, but compare it to what it is mostly doing in a httm -d -R ~
run, stat
all the files in a directory, then multiply that by the number of snapshots of that dataset, and some multiple of additional percentage of deleted files (and don't necessarily divide by the number of your cores, each directory search is a single thread event). Quick example: On one system find ~/program/httm -exec stat {} \;
takes 6s, httm -d -R ~/program/httm
takes 42s with 300 snapshots and 4 low power cores. Now imagine a system that has deleted 60,000 files from a cache, instead of 20,000. Now run an httm
search on each deleted file/dir. Does your performance seem strange? I'm not certain. This could just be inherent to the problem (disks are slower than CPUs, metadata lookups are slow).
As it stands, this feels more like an unintuitive ticker, rather than broken behavior or a real performance problem to me. Maybe the solution is "Find another trigger to make the indicator tick".
httm
is a system punisher, because there is lots of work to do. That httm
uses all 32 threads, just as easily as 4 threads, is awesome. Whether there is additional performance to be gained, as we can see, sure possibly. The question is: At what cost? Does it matter in the 95% case?
But, in order to answer those questions, I'm going to need more info about any problematic directories. So, if you wish to take this further, you'll need to ride shotgun. I'd suggest you take a week and keep playing with httm
. Find me something I can reproduce consistently and I'll take another look.
Ubuntu 22.04 - AMD Ryzen 9 5950X 32GB RAM
When running "httm -d -R ~" CPU spikes and PC becomes unresponsive.
installed from httm_0.17.0_amd64.deb