search

ucrawler / nntplexer

Everyone is a cowboy here.
48 stars 14 forks source link

Slow connection to usenet servers and high cpu usage #5

Open MythodeaLoL opened 2 months ago

MythodeaLoL commented 2 months ago

I tried to perform some usage tests, and I only managed to do so using nzbget, but it was incredibly slow to connect to the configured usenet servers. I couldn't even use another application. When the articles were being downloaded, I noticed a high CPU usage. Perhaps an optimization in the download and connection could help improve this.

I had high hopes of being able to use this now, to fix the problems of missing articles that I'm having when using a specific program that doesn't support what's supported here. Unfortunately, I couldn't. But that doesn't mean it's not a great project, and I'll keep following its growth.

I noticed that some things aren't complete yet, like STATS. This is understandable since it's just starting out, and I'm rooting for it and looking forward to seeing how everything turns out.

ucrawler commented 2 months ago

Set all backends to prio 0. This will give the highest performance but articles will only be tried once.

To have an article tried again in (sequential order!!!) set a backend to prio 1 or higher.

I think you expected to try all backends simultaneously?

MythodeaLoL commented 2 months ago

I think you expected to try all backends simultaneously?

no, I tested that too, but I tested it by setting priority 0, 1, 2

ucrawler commented 2 months ago

Look, adding a proxy between the client and provider adds another hop for the roundtrip. When you use multiple backends with different priority the effect is pretty much exponentially.

100 MB/s -> 25 MB/s -> 6 MB/s etc etc

This is by design to keep the open connections as low as possible. Its primary use is to have many clients connected simultaneously. If your usage is only 1 client, you might as well use sabnzbd...