wget2 stuck

[Irbis01]

i tried to download http://f4se.silverlock.org from terminal or python(os.system). My wget2 sometimes gets stuck to 5-10 minutes and does not write anything to the log. In the end it loaded succesfully. Sometimes this time can be 10-15 minutes command: wget2 --no-cookies --mirror -e robots=off --max-threads=10 -D f4se.silverlock.org -P /home/user/download/f4se http://f4se.silverlock.org/

Most often this site is downloaded in 2 minutes. most likely there is a timeout wait, but it takes a very long time. Perhaps it should be shortened with --timeout or is it still related to some kind of error? The process just hangs and doesn't log anything.

[rootkea]

Which wget2 version?

Wget2 1.99.1 on Ubuntu successfully downloads without getting stuck.

...
HTTP response 200 OK [http://f4se.silverlock.org/archive/f4se_0_06_21.7z]
Downloaded: 86 files, 29.29M bytes, 1 redirects, 0 errors

Most probably, it was a server issue.

PS - BTW, this makes me think we should show time taken to complete the wget2 command with average download rate maybe. Something like "Downloaded: 86 files, 29.29M bytes, 1 redirects, 0 errors, in m minutes, with average download speed: s Kbps"

[Irbis01]

OS: Ubuntu 20.4
command: 
wget2 --version

result:
GNU Wget2 1.99.1 - multithreaded metalink/file/website downloader
+digest +https +ssl/gnutls +ipv6 +iri +large-file +nls -ntlm -opie +psl +iconv +idn2 +zlib +lzma +brotlidec +bzip2 +http2 +gpgm

[Irbis01]

I compare the number of downloaded files with the already downloaded site and their number is the same, but wget2 does not complete its work. freezes occur irregularly. This is very similar to the absence of a response from the server, but all the files are already downloaded

[rootkea]

@Irbis01 Run your command with -d to get the debug output. Copy and paste the complete o/p in a text file and upload that file here.

Also, do mention exactly where wget2 is stuck for you e.g. which line is on the screen when wget2 is stuck, for how long etc.

[Irbis01]

Thanks. I am running wget from subprocess.Popen Python. I will try to repeat the same event in the terminal with the -d parameter

[Irbis01]

I tried to catch the stuck several times, but it only succeeded 3 times out of 15:

1. from terminal
2. from python code with subprocess.Popen

sometimes site downloaded quickly: about 10-30 second. At some times (when the log files are made) it took about 1.5 minutes and I stopped the work

and i found 1 think: wget2 create // from command with the / at the end for command:

 -P /home/user/test/wget_tester/

wget2 create this path:

/home/user/test/wget_tester//obse.silverlock.org

[Irbis01]

commands in terminal:

wget2 -d --no-cookies --mirror -e robots=off --max-threads=10 -D obse.silverlock.org/ -P /home/user/test/wget_tester/ https://obse.silverlock.org/ -o /home/user/test/wget_tester/obse_2.log

commands in python:

command = 'wget2 -d --no-cookies --mirror -e robots=off --max-threads=10 -D obse.silverlock.org -P /opt/project/worker/download/10 https://obse.silverlock.org/ -o /home/user/test/site_test/wget/logs/obse.log'
command_list = command.split(' ')
proc = Popen(command_list, stdin=PIPE, stdout=PIPE, stderr=PIPE)
proc.wait()

f4se.log obse.log obse_2.log

[Irbis01]

What about this info? i catch this stuck sometimes

[rootkea]

I'm on a metered connection. So every download for me takes while. ;)

Last time this command took 5+ minutes for me and you think 1.5 minutes is stuck! So, I can't reproduce this speed/stuck issue at my end. At least not at the moment. But maybe others can?

Just hang in there and maintainers will reply. :)

[rockdaboot]

Since GNU Wget2 1.99.1 we fixed a bunch of bugs. With the latest version from git master this issue is not reproducible. I hope we soon find some time to make a new release.

[Irbis01]

Thank! we will wait the release

[rockdaboot]

The new release v2.0.1 is out: https://gitlab.com/gnuwget/wget2/-/tags/v2.0.1

[alexeyrakov01]

Thanks! We will try