search

MalloyDelacroix / DownloaderForReddit

The Downloader for Reddit is a GUI application with some advanced features to extract and download submitted content from reddit.
GNU General Public License v3.0
498 stars 47 forks source link

What is considered normal memory load during operation? #208

Open T-prog3 opened 3 years ago

T-prog3 commented 3 years ago

What is considered normal cpu and memory load during the softwares operation? My cpu useage go between 11% - 60% and memory over 2000mb.

zacker150 commented 3 years ago

Depends on how fast you're downloading and how good your CPU is. In general, high speed downloads take a lot of CPU since it's rapidly copying data from your memory to your disk.

MalloyDelacroix commented 3 years ago

I haven't done benchmark testing to see what the normal really is. Based on my experience, the CPU usage doesn't seem unreasonable depending on your hardware and how many extraction and download threads you are running. I will say though, the memory usage does seem high. I've never seen a number close to that. But, this would also depend on how much is being downloaded and to some extent the thread count settings.

If these numbers are too high, you could reduce the download/extraction thread counts or break large download lists into several smaller lists that are run sequentially.

T-prog3 commented 3 years ago

I use 6 download/extraction threads on an Amd Ryzen 5 1600 Six-Core Processor and 16gb memory. And it seems to be some cases where the memory go through the roof. I've done about 15 test runs now with diffrent sizes of lists and most of them keeps the memory between 350 - 500mb.

Yesterday when i reached over 2000mb i downloaded about 50 users. And today i tested with list sizes of 150 - 300 users to see what happens. In 1 of 15 runs the software went over 500mb. I can't tell what happend but it ended with system slowdown and an Out-of-memory crash. And it left me with lots of unmerged mp4 and mp3 files.

MalloyDelacroix commented 3 years ago

Does this usually happen toward the end of a download? And also when you are downloading a lot of video posts?

T-prog3 commented 3 years ago

I'm not able to say if it is close to the end cause the ui doesn't tell who its current download is. And the green bar is always at 96-98%. But i've noticed that it started to happen somewhere about 20-30 minutes into the process and never from start.

I believe that video posts may have something to do with it but i'm not sure. All i can say is that i normaly don't encounter videos that frequently and just happened to have lots of them at an unmerged stage after the out-of-memory crash.

Edit: Just did a run and 18 minutes into the download there started to be some obstacles where the download speed started to decrease and memory consumptions went up more and more. It appears to be some larger files who gets downloaded and halts the speed until its done. But during that halted time the memory only goes up and is not freed again when it's done.

Edit2: I did stop the previous download because of high memory consumption. But wanted to try again and get any remaining files so i changed my list settings from Last Post Date to -> Custom Date Limit: 2005 so that i'm not stopped at the first already downloaded post. But this setting made things way worse. I'm 15 minutes into the download with only 5 files downloaded and it have eaten 1900mb memory before i've reached the destination of the previous ending.

The software seem to be collecting alot of garbage in memory during the run and wont free any up until the very end so in most cases where there is more than some hundred users with lots of downloads it will probably crash the whole system every time.

MalloyDelacroix commented 3 years ago

This sounds like there may be a memory leak in the multi-part downloader. If you could attach your log file, it would help me diagnose the problem.