search

Tensai75 / nzbrefresh

Cmd line tool to re-upload articles that are missing from providers
MIT License
15 stars 4 forks source link

Extremely slow operational speed #6

Open gyurix opened 1 month ago

gyurix commented 1 month ago

The program is extremely slow, I am measuring 51.81MB/s speed only on Windows with 4 providers from 4 different backbones. @Tensai75 can you please improve the programs performance, so it would work better on huge consolidated NZB files? I would like to see ideally at least 100x the current speed, that way making it run through my 21TB nzb file throughout 1 hour instead of 121.

nzbrefresh --provider providers.json --debug all.nzb
NZBRefresh v0.0.0-alpha-3
Starting segment check of 30283012 segments
Checking segments :       (62172/30283... [----------------------------------------]     [15m 1s|121h 38m 15s]
MythodeaLoL commented 1 month ago

hi @gyurix, i'm running with newshosting(99) and thundernews(49)

I merged all the nzbs I found(8GB), 66,957,079 segments. 12h to finish. the final time is uncertain because when repairing, it may increase the time.

If you are not running on a dedicated machine with good hardware or network resources, I suggest you try this. Don't forget to set the host provider to the same location/country as the machine.

I am running from linux x64.

a 21 TB????? nzb file seems like overkill, I would suggest splitting it up, and running it in parts, or you could buy extra providers and run each part of the nzb file in parallel.

I don't think you can optimize it any further than that. But good luck, if you do let us know.

gyurix commented 1 month ago

21TB was the referenced content size of the nzb, as I see you have double of that. Maybe it's just the Windows version of the application which is so slow...

Tensai75 commented 4 weeks ago

The bottleneck is the provider with the slowest speed as a combination of the lowest connections and the highest response time. My tests generally resulted in average response times for the STAT command of around 50 ms. So if the slowest provider has 100 connections and an average response time of 50 ms, the program can process around 2000 segments per second in best case (if no upload is required). The faster the response time is, the more segments can be processed. But if the slowest provider has only 50 connections and the same average response time (or worse), the overall speed drops to 1000 segments/s (or less). So, the only way to increase the overall speed is to use more connections and I might add at some time the ability to use more than one account from the same provider, as suggested in this issue here: #7

In your case, using the windows version, it could be the same problem I am having. On my Win10 PC, for some reason, nzbrefresh cannot establish more than 100 working tcp connections. If more connections are opened, the packets are dropped and the connections are closed by the operating system. I have this issue only on Windows and nzbrefresh is working perfectly fine on my Ubuntu PC. To check this, use the -d option and check the log file for NNTP Pool Errors such as EOF or other errors indicating that a connection has been closed. Or use a smaller NZB file and check the number of connections used by each provider at the program end. If not all providers have used at least 90% of the maximum number of connections specified in the config, then there is probably a problem with your connections as well. Although I have been trying to troubleshoot this problem for a long time, I still have no idea what is causing it. But I'm pretty sure it's not a problem with nzbrefresh, but rather a system limitation or maybe even a hardware problem.

gyurix commented 4 weeks ago

@Tensai75 Is it possible to queue up multiple segment stat requests with a single NNTP packet sent and then get a response from the provider to everything asked all at once, that way making that 50ms time apply for multiple segments checked not just one per connection? so i.e. 100 connection, to each we send 100 article ids at once, so in total requesting stat for 10k articles not just 100 and then wait for response, can it work like that or you tested that already and it's not possible to do that?

Tensai75 commented 4 weeks ago

@Tensai75 Is it possible to queue up multiple segment stat requests with a single NNTP packet sent

No, unfortunately this is not possible. Each STAT request is an individual request. They cannot be combined. See: https://datatracker.ietf.org/doc/html/rfc3977#section-6.2.4