search

rockdaboot / wget2

The successor of GNU Wget. Contributions preferred at https://gitlab.com/gnuwget/wget2. But accepted here as well 😍
GNU Lesser General Public License v3.0
567 stars 76 forks source link

wget2 of large files extracted from a url using Win10 OS causes blue screen of death and computer shut down #354

Open k-blenman opened 1 week ago

k-blenman commented 1 week ago

System Information: Win10 OS v10.0.19045 Build 19045; Lenovo; x64; Intel Xeon CPU E5-2630, 32GB RAM, GPU NVIDIA Quadro K620

I am using wget2 to download over 300 files with a 3GB - 7GB file size for each file from a specific url. I am saving the downloaded files directly to a Western Digital external hard drive. The process starts to download the files and runs fine for ~20 mins to 1 hour then the Win10 blue screen of death pops up and shuts the computer down. The process happens too quickly for me to see if there is an error message before it shuts down. Especially since its time to shutdown is not consistent. Note that there are no error messages at the beginning or throughout the run that I have noticed. However, I have noticed that it seems to try to download 4 to 8 files at one time, which I believe may be the problem given the file size. Is there any way to force wget2 to finish downloading one file before it starts downloading another (i.e., only download one file at a time). Here is what I am running:

wget2 -r -nH --cut-dirs=2 -l 2 --execute robots=off url

Any suggestions would be greatly appreciated.

Thank you.

rockdaboot commented 1 week ago

Try with --max-threads=1

rockdaboot commented 1 week ago

But tbh, a blue screen of death possibly means that your disk is broken and/or your file system needs a repair.

rockdaboot commented 1 week ago

Possibly start with checking your RAM.

k-blenman commented 1 week ago

Thank you so much for the suggestions. Please note that the Hard Drive is in perfect working condition. It is used frequently with no problems. The RAM is also an unlikely culprit as it is used to process these and even larger files all the time.

I have found similar issues, not blue screen of death, but shutting down or freezing of the software tool itself when trying to process these types of large size files in parallel.

I will let you know how it turns out.

k-blenman commented 1 week ago

So far so good . . . Thank you so much for the --max-threads=1 suggestion. It seems to have solved the problem.