Closed NaPurHab closed 2 years ago
@NaPurHab
OK, 1024 is an insane number (unless you're connected to 1024 trackers at the same time, which I'm going to guess you are not), on my system it's set at 25 (which I think is the default) and I've never seen that message. The advice to increase concurrency as the solution to this problem may be... overly optimistic.
What do you see in the Tools > Statistics > Tracker tab view that's mentioned in the message box? The graph(s) there should give a better picture of the type of announce lag you're experiencing, which sounds like it may be caused by something like an unresponsive/slow-to-respond tracker, or perhaps too many active downloads managed by the same tracker? (What kind of settings do you have in Options > Queue for things like min/max simultaneous downloads, max active, etc.?)
My biggest issue at the moment is the RARBG Trackers showing as offline. So I have added trackers_best (20 trackers)
from ngosang/trackerslist. That list updates daily. Entry looks like this:
Here is the Update Lag, Queue and the System TAGS. Currently I have 5 downloads that are not moving. (added as 4th image below). As to the Queue settings, I have it set to 0, so no upper limit for active torrents. Best I usually get is around 80 seeds active. If I set it at say 84, I get mid-20s for seeds. I'm half wondering if the Ukraine Conflict is involved. Perhaps Russia is upto something on the Internet. I've noticed it is harder to get Trackers to connect since early in the year.
@NaPurHab Hmm... so it's scrapes that are lagging.
What do you have in the Options > Tracker > Client settings? (most especially the Scrape, Protocol, and Override Options sections.)
@ferdnyc Her you go ...
I'd probably start with turning off "Scrape torrents that aren't running", personally. If you have a significant number of inactive torrents (and it seems you do, specifically all of the torrents in various "Queued" states), there's not a lot of value to scraping them all. (Not if it's causing issues, anyway. Which seems to be the case.)
"Manage tracker activation to reduce load" could also possibly help (though I don't know if that will affect scraping at all).
Also possibly related: Two minutes is an incredibly long connect timeout. With that kind of window, every attempt to contact an unresponsive tracker is going to potentially delay a subsequent connection to a working tracker by up to 120 seconds. I'd probably drop that way, way down — if a tracker isn't able to respond to a connection attempt within a few seconds, even if the connection would eventually succeed, it's probably so overloaded (or having other issues) that you're better off just bailing on it and moving to the next one in the list. The whole point of having multiple trackers for each torrent is that connecting to any one of them is unimportant — there's always the next one on the list. Short timeouts ensure that busy trackers or ones that are having network connectivity issues can be treated the same as offline trackers, and passed over in favor of more responsive ones.
Personally, I use 15 seconds for the connect timeout, 35s for the read timeout, and a concurrency of 25. (Along with not scraping inactive torrents.) My stats graphs always look about like this:
@ferdnyc Ok, I am getting this again....
Tried the numbers you suggested, here is what it looks like: Notice I didn't go all the way down to 25, I used 128, down from the 1024 before. But it still tells me to increase the concurrency. Oh and give you an idea of how much I am seeding and queue settings:
That should give you an idea of my LOAD. I have seen BiglyBT reach upload speeds over 250 Mbit/s upload, though more typical is 120-160 Mbit/s. In the last few months, in spite of new torrents on RARBG containing usually only their own Trackers. Those trackers are showing as Error: Offline (No data received)
or Error: Failed (Connection timed out: connect)
. So now upload performance has dropped and has a range is 20 Mbit/s - 70 Mbit/s. And the fun part is I have 500/500 Fibre, and using a program that can download video from various sites, that I just got. It hit a high of 360 Mbit/s. Reported in both the program, and on my Network Meter gadget.
The Network Meter has higher numbers as it shows Overhead and other traffic. And you can see the Distributed Database show 462k users. I miss the days when it was 1.5m users.
So the system is still asking me to increase the concurrency. Easy to see why I had it at 1024. It just keeps asking if I have it lower. And at 128 I still have it higher than the 25 you suggested. My goal with my system is to Seed as much as possible as fast as possible. Though I am currently far from using my 500/500 Fibre to a high load.
So the system is still asking me to increase the concurrency. Easy to see why I had it at 1024. It just keeps asking if I have it lower.
Oh, yeah, no, I totally understand that. The message probably needs to be changed, the assumption that lag issues are caused by a lack of concurrency is clearly incorrect.
What do the Tracker graphs look like, with the updated settings? That's the trigger for that message, ultimately: The lines on the lower-left "Update Lag" graph rising too high.
In your previous screenshot, it showed quite a bit of lag on Scrape attempts. On my system, no lag ever seems to register there under any circumstances.
Now, you have much higher activity levels in BiglyBT than I do, but you also have a much better connection. (I'm going to assume a faster machine as well, because I'm running BiglyBT on a 2-core Athlon 64 box from like 2007.) Which is why I was hoping the options changes would at least improve the lag stats.
If there has been improvement, then at least we're moving things in the right direction and can try to improve them more. But the way to evaluate the effects of any changes would be to look for a drop in the lag shown on that graph. I hope to see that turning off inactive scrapes and lowering connection timeouts helped somewhat. If the changes had NO effect, then it means I'm on the wrong track with that, and there must be some other cause entirely that we'll have to try to understand.
I definitely don't buy (based on how the lag is manifesting) that it has anything to do with concurrency, despite what that error dialog suggests. (Especially when it's scrape lag, something that would have nothing to do with "concurrent announce tasks", because scrapes are precisely not announces. In a sense they're the opposite of announces, or at least an alternative to them.)
IMHO the lag warning message, while possibly itself useful and valid, is at a minimum misrepresenting the problem, by suggesting that concurrency is the sole cause or that raising it is the sole solution. That message could be a decade or more old, and may still be relevant if someone's lowered their concurrency to a single-digit value, but clearly there are other ways to incur lag that it fails to account for.
@ferdnyc
What do the Tracker graphs look like, with the updated settings? That's the trigger for that message, ultimately: The lines on the lower-left "Update Lag" graph rising too high.
Now, you have much higher activity levels in BiglyBT than I do, but you also have a much better connection. (I'm going to assume a faster machine as well, because I'm running BiglyBT on a 2-core Athlon 64 box from like 2007.) Which is why I was hoping the options changes would at least improve the lag stats.
Yah, I have an Intel i7-8700k 3.7Ghz, 32GB Memory, 69TB Storage (10 internal, 1 USB Drive), 500*500 Fibre (not just to the Neighbourhood, my Apt Building was upgraded with Fibre to each unit. Only wire is Cat5 from my router to my computer.) Here is what my CPU load looks like with BiglyBT and a couple of Browsers running (Edge & Edge Beta).
IMHO the lag warning message, while possibly itself useful and valid, is at a minimum misrepresenting the problem, by suggesting that concurrency is the sole cause or that raising it is the sole solution. That message could be a decade or more old, and may still be relevant if someone's lowered their concurrency to a single-digit value, but clearly there are other ways to incur lag that it fails to account for.
You may be on to something. But that is what beta testing is about. Finding issues. I had an issue with the Scheduling back when it was Vuze. And using my suggestions Parg was able to make the Schedule work to my needs. And probably lots of other peoples too. Basically, I wanted to be able to Schedule Downloads to only Download during certain hours, unless over-ridden. I was on DSL 15/10 with a 200gb/month download cap. But with no Upload Cap, and an exception of Unlimited Download from 2am to 8am. Hence all the changes to the scheduling that are still there a decade and a half later. I missed the split and didn't get BiglyBT until around 1.5.0.0. Wish we could push a notice out to all the people still using Vuze.
Anyway, I am appreciating all your help. Maybe Parg will see your comment and consider an overhaul of Old Code with New Understanding.
Anyway one last image, my current upload speeds. Remember I used to usually get 120-140Mbit/s as average uploads with good days going upto 160-180, and have seen on rar days uploads peaking around 240, but that is rare. I'd just be happy to get my upload speeds back to the 120-140 range.
You have 1838 announces scheduled - for example 300 active torrents each with 6 trackers and all forced active for whatever reason. Or 1838 torrents with 1 tracker. Or 1 torrent with 1838 trackers. I don't know. Of these 128 announces are actively running. 953 are pending - meaning they are ready to run but can't because all 128 threads are busy. You have a lag average of 534 seconds. This means that on average each announce is taking 534 seconds longer than it should. So you have some trackers that are responding very slowly and causing things to queue. It doesn't matter how fast your network connection is, if BiglyBT sends an announce request and the TCP socket/UDP "connection" takes 500 seconds to give a reply/timeout, things are going to get queued.
That's actually kind of fascinating!
If you recall, in your previous screenshot of the stats, there was a ton of Scrape lag, but tracker announces were barely a blip. Now, we've succeeded in clearing up the scrape lag (yay), but for some reason your announce lag has shot up to (frankly) mind-boggling heights.
Did you happen to activate "Manage tracker activations to reduce load", in the options? I was pessimistic about that setting having any effect, before — because we were looking at scrape lag — but now seems like the exact sort of situation it's designed to help with.
Sorry for the long delay, bit under the weather. As to your question, NO, I did not enable Manage tracker activations to reduce load
.
It is enabled by default so you have disabled it.
I keep getting the following warning:
I have slowly increased the concurrency, and it still keeps happening. It is currently at the MAX of 1024. Any ideas how to solve this?
Java 1.8.0_333 (64 bit) Oracle Corporation c:\program files\java\jre1.8.0_333
SWT v4942r22, win32, zoom=100, dpi=96 Windows 10 v10.0, amd64 (64 bit) B3.1.0.1_B07/4 az3