search

rdavydov / Twitch-Channel-Points-Miner-v2

A simple script that will watch a stream for you and earn the channel points.
GNU General Public License v3.0
1.11k stars 326 forks source link

High amount of DNS Requests from Twitch Miner (Container) #369

Open Cebrain opened 9 months ago

Cebrain commented 9 months ago

Describe the bug

I use a DNS Filter and the Twitch Channel Miner Docker is the device in my network with the most requests. All my other devices have only a small part of the requests. Its no big issue but i thought maybe someone can check this. Is it possible that the miner is not caching DNS Entrys? The 2 most quested Domains are "www.twitch.tv" and "gql.twitch.tv" Which are alright but i dont understand why its necessary all 2 Minutes.

Steps to reproduce

Only let the Miner run in the container

Expected behavior

That the client is caching the DNS Entrys and only requests after an TTL period

Operating system

Linux - Container

Python version

3.11.1

Miner version

1.7.7

Other relevant software versions

Linux Distribution and Twitch Channel Point Miner runs in Docker, latest version.

Logs

-

Additional context

Recommendation from @T0biii https://github.com/Tkd-Alex/Twitch-Channel-Points-Miner-v2/issues/742#issuecomment-1734589095

maybe https://pypi.org/project/requests-cache/ could mitigate this
for experation see: https://requests-cache.readthedocs.io/en/stable/user_guide/expiration.html
NoxRare commented 9 months ago

Pretty sure twitch serves their content from different AWS servers, thus if you start to cache the dns records you may impact the bots performance, there is no need modify this at all... This is not impacting anyone currently and unless youre actually being negatively impacted by the dns requests this should not be changed at all.

Cebrain commented 9 months ago

As i understand DNS, the DNS record gets per Default an TTL which tells the Client (or local DNS Server (Router)) when the entry is old and needs to be refreshed. So the local DNS Server (Router, PiHole, MS DNS Server, etc) that is providing the entry to the miner, is caching the entry with an TTL already and this is not causing any problems at the moment.

Many services are provided by AWS and every OS is caching DNS entrys and this is not causing any problems? (i know the Miner is different then a default client but still) I could think of an performance benefit, because the Miner doesn't need to connect the DNS Server every time.

Edit: To calrify, i dont have any impacting problem by this design choice, still i think the performance would benefit.

rdavydov commented 9 months ago

AFAIK, it is not an app's job to cache the DNS records but OS's.

rdavydov commented 9 months ago

And we also have the websocket connections. requests-cache would not help with those.

I use a DNS Filter and the Twitch Channel Miner Docker is the device in my network with the most requests.

Could you please support this statement with some data? How exactly did you determine it?

NoxRare commented 9 months ago

Your router is not a DNS server, your ISP usually has its own DNS server that you connect to by default. Personally I have a DNS server that I use to block ads network wide and I've had no issues with a large amount of requests, I host over 12 instances for my friends and family using Pterodactyl as panel and DNS requests are rare occurrences at best. They are not even on my top list lol.

EDIT; And even my DNS server doesnt actually fetch the info itself.. it just blocks requests that I dont want and forwards the requests to Google & Cloudflare DNS. Even if the bot is sending a large amount of requests this will not hit any significant limit, this should not be something to be worried about tbh. Unless youre using your ISP and their limit is unreasonable, just use any other DNS server. There are plenty to choose from that are good. My samsung TV sends 4 DNS requests to different samsung endpoints every 3 mins, this is a common and expected behavior...

EDIT 2; If youre using Pihole and the DNS requests is cached the request is still not being sent out? The request is staying in the local network, meaning no outside connection is made either way and there is exactly zero performance hit either way you look at it..

Cebrain commented 9 months ago

AFAIK, it is not an app's job to cache the DNS records but OS's.

You are completly right and sorry if i didnt wrote is explicit in the headline. I use the docker container, maybe because of that this is a missunderstanding. I cant modify the container os for this purpose.

About providing some log and data, i will provide the next day.

Cebrain commented 9 months ago

Your router is not a DNS server, your ISP usually has its own DNS server that you connect to by default. Personally I have a DNS server that I use to block ads network wide and I've had no issues with a large amount of requests, I host over 12 instances for my friends and family using Pterodactyl as panel and DNS requests are rare occurrences at best. They are not even on my top list lol.

EDIT; And even my DNS server doesnt actually fetch the info itself.. it just blocks requests that I dont want and forwards the requests to Google & Cloudflare DNS. Even if the bot is sending a large amount of requests this will not hit any significant limit, this should not be something to be worried about tbh. Unless youre using your ISP and their limit is unreasonable, just use any other DNS server. There are plenty to choose from that are good. My samsung TV sends 4 DNS requests to different samsung endpoints every 3 mins, this is a common and expected behavior...

EDIT 2; If youre using Pihole and the DNS requests is cached the request is still not being sent out? The request is staying in the local network, meaning no outside connection is made either way and there is exactly zero performance hit either way you look at it..

Sorry if i hit a spot. I only wanted to provide some enhancement and i write it again i use the docker container and it would be great if the docker os would cache the entrys.

Yes you are right, the performance impact is probably neglectable for the network. Still it is unecessary traffic if the docker os could cache it.

I dont wanna go in detail about DNS servers now, but in a default network the router is configured on most devices as DNS server and the router sends the request to the ISP.

T0biii commented 9 months ago

@Cebrain maybe try this on OS Level: https://stackoverflow.com/a/71192582 "Now you can enable systemd-resolved with " 

systemctl enable systemd-resolved

maybe then Docker uses also the cache

Cebrain commented 9 months ago

@Cebrain maybe try this on OS Level: https://stackoverflow.com/a/71192582 "Now you can enable systemd-resolved with "

systemctl enable systemd-resolved

maybe then Docker uses also the cache

Thanks for the suggestion and pointing in the right direction. For the moment i was possible to install systemd and enable resolved but it is still not caching the requests as i see in the logs. I dont know how to configure the linux/container system to use resolved at the moment but will try it in detail later and probably find something in the net ;)

apt update
apt install systemd
systemctl enable systemd-resolved
Cebrain commented 9 months ago

And we also have the websocket connections. requests-cache would not help with those.

I use a DNS Filter and the Twitch Channel Miner Docker is the device in my network with the most requests.

Could you please support this statement with some data? How exactly did you determine it?

Example screenshot from my DNS Server over the last days. Miner is at top with requests and all other devices are below censored (Computers, Smartphones, Server, etc.)

https://i.imgur.com/3N0X2cX.png

NoxRare commented 9 months ago

Try setting your network-mode to bridge, docker docs: https://docs.docker.com/network/#dns-services You may be using a custom network, are you using docker compose or command? If youre using compose, is it possible you are using portainer? Portainer makes a custom network for each "stack" unless specified otherwise.

By default, containers inherit the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that attach to the default bridge network receive a copy of this file. Containers that attach to a custom network use Docker's embedded DNS server. The embedded DNS server forwards external DNS lookups to the DNS servers configured on the host.

Cebrain commented 9 months ago

Try setting your network-mode to bridge, docker docs: https://docs.docker.com/network/#dns-services You may be using a custom network, are you using docker compose or command? If youre using compose, is it possible you are using portainer? Portainer makes a custom network for each "stack" unless specified otherwise.

By default, containers inherit the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that attach to the default bridge network receive a copy of this file. Containers that attach to a custom network use Docker's embedded DNS server. The embedded DNS server forwards external DNS lookups to the DNS servers configured on the host.

You are right, i am using a dedicated network / macvlan, didnt know there is a difference with DNS resolvers on the network. As i see it is only the DNS Config, but not about caching i think? I want the container to have an different IP address from the host, so i can set firewall rules. So i think most people dont will have this problem. Will check what i can do ;)