arvidn / libtorrent

an efficient feature complete C++ bittorrent implementation
http://libtorrent.org
Other
5.28k stars 996 forks source link

DNS cache improvement suggestions #7381

Open nasbdh9 opened 1 year ago

nasbdh9 commented 1 year ago

libtorrent version (or branch): all

platform/architecture: unrelated

compiler and compiler version: unrelated

prerequisite environment for problems to arise: tracker server uses CDN (such as Cloudflare) or tracker server domain with multiple IPs

happens when the problem arises: some ip returned by dns when resolving tracker servers are interfered with by the network, causing tracker to be unavailable

Current solutions to this problem: wait 1200 seconds (20 minutes, default) for the DNS cache to expire after the re-resolution to get another IP available and the tracker to resume normal operation

or still get an unavailable IP (interfered with, or unavailable itself) and then wait 1200 seconds and try again to resolve the new IP

Ideas for improving DNS cache: When the tracker status is unavailable, immediately discard the DNS cache for the wrong tracker, then re-resolve and try to re-report. (A publicly settable value may be needed to limit the number of retries, and to wait X seconds after X attempts to try again if it is always unavailable)

A time-limited blacklist can be set for erroneous IPs, and unavailable IPs will be in the list for a period of time (default might be 1 hour? or a value that can be set)

arvidn commented 1 year ago

Do I understand correctly that the scenario you're describing is when you do get a response from the DNS, you just get an IP that's invalid in some way, yeah?

The error message you posted in the other thread suggest that the IP is invalid in that it's in some reserved IP range, not valid for the public internet. Is that right?

I don't think it's completely safe to assume that's an error. It's fine to run a tracker on a local network for instance.

nasbdh9 commented 1 year ago

Do I understand correctly that the scenario you're describing is when you do get a response from the DNS, you just get an IP that's invalid in some way, yeah?

Yes

The error message you posted in the other thread suggest that the IP is invalid in that it's in some reserved IP range, not valid for the public internet. Is that right?

No, this is not a reserved IP. It is just temporarily unavailable due to network failure or interference.

I don't think it's completely safe to assume that's an error. It's fine to run a tracker on a local network for instance.

Not local tracker

arvidn commented 1 year ago

I see. So the DNS cache is fine. There is no need to invalidate it, the tracker just need to be retried (using the IP its name resolved to the first time).

Is that right?

nasbdh9 commented 1 year ago

using the IP its name resolved to the first time

not using the IP its name resolved to the first time, because he has made the tracker status unavailable (IP disturbed/faulty)

the tracker just need to be retried

Right, when the tracker status is unavailable let it re-resolve so that it can get another IP instead of continuing to try to report using the existing cache.

If the IP is available after re-resolution the tracker will resume working, if it is still unavailable then try to continue resolving. (it is better to provide a setting of X seconds to wait X seconds before resolving if it is still unavailable after X attempts, to avoid DNS Query Storm)

arvidn commented 1 year ago

Ok. I was trying to distinguish between re-resolving and re-connecting.

A tracker that can't be connected to; I don't think it's safe to assume re-resolving would make a difference.

Typically, if a hostname has multiple IPs you receive multiple ones when resolving. If one fails, libtorrent will try the next IP.

Are you sure the problem is that the IP doesn't work and that you get a different IP when re-resolving?

nasbdh9 commented 1 year ago

Are you sure the problem is that the IP doesn't work and that you get a different IP when re-resolving?

yes, will be useful if DNS returns a different IP after re-resolving (tracker using CDN)

But after discussing it with other users, I think I may have complicated the issue. Back to the original idea, a "button" to immediately refresh the DNS cache would be enough. This even if system hosts are changed, the IP of the tracker can be updated immediately.

Roardom commented 1 year ago

Some private trackers tend to put their announce behind multiple proxies (single domain name resolving to multiple ipv4 addresses). Sometimes some of these proxies temporarily go offline, and it is expected for the client to simply use another of the IPs. Do the recent DNS cache changes prevent this from working?

arvidn commented 1 year ago

it is expected for the client to simply use another of the IPs. Do the recent DNS cache changes prevent this from working?

No, and the DNS cache isn't really involved in this logic. It just caches all the IPs a name resolves to.

arvidn commented 1 year ago

Are you sure the problem is that the IP doesn't work and that you get a different IP when re-resolving?

yes, will be useful if DNS returns a different IP after re-resolving (tracker using CDN)

did you confirm that the first DNS lookup only returned a single IP address? did you confirm that the first IP you received was failing? how did you confirm it and what error message did you get? I would expect either "connection refused" or "host unreachable". did you confirm that a second DNS lookup gave you a different IP address? (was it still just a single IP?) Typically DNS servers would cache the response for some time, so you would presumably have to wait the TTL time before trying again. Does this hostname have a low TTL set?

Back to the original idea, a "button" to immediately refresh the DNS cache would be enough. This even if system hosts are changed, the IP of the tracker can be updated immediately.

I would want to understand what the problem is before making any changes.

nasbdh9 commented 1 year ago

did you confirm that the first DNS lookup only returned a single IP address?

No, multiple ip

did you confirm that the first IP you received was failing?

yes

how did you confirm it and what error message did you get? I would expect either "connection refused" or "host unreachable".

skipping tracker announce (unreachable)

did you confirm that a second DNS lookup gave you a different IP address? (was it still just a single IP?)

yes. even if dns does not return other ip, my can edit the system hosts to use other ip of cdn service provider But currently there is no way to refresh libtorrent DNS cache immediately after edit system hosts

Typically DNS servers would cache the response for some time, so you would presumably have to wait the TTL time before trying again. Does this hostname have a low TTL set?

Except for the system and program DNS caches, the DNS servers on the LAN do not have caching enabled, so each resolution on the LAN after the system and program caches expire will re-query the upstream public DNS servers

ValeZAA commented 1 year ago

and it is expected for the client to simply use another of the IPs.

Nope.

That is not how standards work: when you get multiple "A" v4 IPs from one domain you need to perfectly randomly select one of them. You are not allowed to invalidate one of them. Now I don't quite remember what should happen if one of the IPs is bad and does not respond to TCP handshake, I wonder should you request DNS again... but I actually think it is just making the domain bad.

The whole balancing /round-robin standards are very complicated.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Zylsjsp commented 8 months ago

There is an memory leak issue in qbittorrent which may be related to the dns caching and establishing tracker connection. As mentioned in https://github.com/qbittorrent/qBittorrent/issues/20117, my fake-ip on clash tun seems to be related in huge memory consumption in connection with trackers. Should i open another issue?