1000s of DNS lookups to api.glowmarkt.com

smithbill17 commented 2 years ago

I use Pi-hole (see https://pi-hole.net/) on my LAN to filter out excessive internet adverts and also as my DHCP & DNS lookup server.

Sometime after installing this integration, I noticed many 1000's of DNS lookups to api.glowmarkt.com in my Pi-hole dashboard. There are 30+ lookups every 1second.

To see these lookups, you would need to setup Pi-hole or some other network device to receive & forward DNS queries. I have attached a screenshot.

Surely 30+ DNS lookups every 1 second is unnecessary and presumably not good for LAN, ISP or Hildebrand servers!?

Home Assistant v2021.10.6 Hildebrand Glow (DCC) integration v0.5.0

Screenshot_2021-10-27-12-59-34-366_com android chrome

ColinRobbins commented 2 years ago

I confirm I am seeing similar too. Investigating...

ColinRobbins commented 2 years ago

Update My test system seems to be behaving, and making only 8 lookups every 2 minutes, which is what I'd expect.
Similar on live. So for my system, that is 8 30 2 requests per hour. Today we are at 15:00 so 15 8 30 * 2 = 7200.
In my logs I see 9000 DNS calls, so not too far out.

ColinRobbins commented 2 years ago

Looking at the glowmarkt DNS entry, they have the TTL set to 5 minutes.

api.glowmarkt.com | IN | A | 154.51.148.100 | 300s | (00:05:00)

This seems low, but doubt the full reason for the issue.

smithbill17 commented 2 years ago

I have rebooted my Raspberry Pi that runs HomeAssistant (in a docker container) and as far as I can see, there are still runs of DNS lookups in Pi-hole. It looks like every 2mins there are 24 DNS lookups in the space of 2seconds - see part of the Pi-hole report below.

2021-10-2716:15:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
-- | -- | -- | -- | -- | -- | --
2021-10-2716:15:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:15:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.2ms) | Blacklist
2021-10-2716:15:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.2ms) | Blacklist
2021-10-2716:15:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.2ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.3ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.2ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.1ms) | Blacklist
2021-10-2716:15:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:15:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:13:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:13:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:13:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:13:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms) | Blacklist
2021-10-2716:13:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:13:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.1ms) | Blacklist
2021-10-2716:13:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (1.7ms) | Blacklist
2021-10-2716:13:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.2ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (1.6ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (1.8ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (2.4ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.5ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (2.3ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.6ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (2.3ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.7ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (3.1ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.8ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (2.5ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.3ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (2.5ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (2.5ms) | Blacklist
2021-10-2716:13:06 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | NODATA (3.6ms) | Blacklist
2021-10-2716:13:06 | A | api.glowmarkt.com | rpi4b-eth0 | OK, answered bylocalhost#5335INSECURE | IP (4.4ms) | Blacklist
2021-10-2716:11:07 | AAAA | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | NODATA (0.1ms) | Blacklist
2021-10-2716:11:07 | A | api.glowmarkt.com | rpi4b-eth0 | OK(cache)INSECURE | IP (0.2ms)

smithbill17 commented 2 years ago

According to my Pi-hole, in the 24hrs till now, there have been 15,749 DNS lookups of api.glowmarkt.com. That's about 13,000 hits more than the next highest permitted domain lookup which is alexa.amazon.co.uk So it is kinda 'orders of magnitude' higher than anything else.

ColinRobbins commented 2 years ago

Thanks for the info. A set of DNS lookups every 2 minutes is what I would expect. The code makes a call for: 1) Pull latest usage data from DCC to Glow 2) Pull latest cost data from DCC to Glow 3) Get current usage 4) Get current cost 5) Get cumulative usage 6) Get Tariff data.

It will do this for Electric and Gas - so 12 "Get" requests.

It seems the code creates a new session for each request, so this might be where the other 12 come from.

An improvement would be to use requests.Session(), but that will take time to figure out how the HA model deals with that.

smithbill17 commented 2 years ago

Okay, it definitely seems to be 24 lookups every 2mins taking ~2secs (in the Pi-hole dashboard log). I'll monitor it for now, but it would be good to reduce the lookups to something sensible as it WAY outstrips everything else on my network. I'm not sure whether it has a perceptible impact on network performance.

ColinRobbins commented 2 years ago

I've researched this a little.

The integration uses the python requests package, which makes a DNS query per-request.

Using a requests.session() might fix it, so made a quick fix, this cuts down the number of calls to one per request (I think as the session is open, so SSL established). I’ll keep, testing this an make a PR in due course as it’s better.

I guess we need to determine how other integrations deal with the DNS lookup issue.

ColinRobbins commented 2 years ago

I have posted a question on this in the wider Home Assistant community, to see what others may suggest:

https://community.home-assistant.io/t/multiple-dns-queries/350765

si458 commented 2 years ago

@smithbill17 daft question, are you using the integration or did you just install it to have a look at it? it might be a 401 error loop if you havent setup the credentials?

smithbill17 commented 2 years ago

@smithbill17 daft question, are you using the integration or did you just install it to have a look at it? it might be a 401 error loop if you havent setup the credentials?

Yes, I'm using it. It's logged in with appropriate credentials & receiving data okay (although it still seems to behave a little oddly just after midnight, but that's a separate issue @ColinRobbins has already looked at). I still get 24 DNS lookups every 2mins.

ColinRobbins commented 2 years ago

I’ve got the 24 lookups down to 12, but am waiting for a week before I issue a PR. I do want to check there is not 401 loop as @smithbill17 suggests. This is a possibility after 7 days of running, when the token expires.

si458 commented 2 years ago

I still get 24 DNS lookups every 2mins.

i think thats divided by 2 so only 12 lookups every 2mins as its looking up AAAA and A together (ipv6 and ipv4)

i see it from my servers time to time, it looks up the IPv6 addresses even though i dont use IPv6

BUT then divide 12 by 2 again (2 services gas + elec) thats 6 requests each

but that would be right as if you look into the raw api https://api.glowmarkt.com/api-docs/v0-1/resourcesys/#/

it needs to call

catchup (refresh readings)
tariff prices
readings for the 30min window
readings for the whole day
resources (to check if you have randomly added extra smart meters, you never know with the magic software fairys)
refrsh token

ColinRobbins commented 2 years ago

That’s is not quite how it works. I have listed the 6 calls the code makes in a post above. 6 for gas and 6 for electric.

xhemp commented 2 years ago

Looks like after the network issue Glow had with their systems last week/weekend ( https://forum.glowmarkt.com/index.php?p=/discussion/comment/540#Comment_540 ), they just posted an update on their forum with the following:

UPDATE from the Glow team

Because some users without our CAD are calling the API much too frequently (it should be once per half hour) we have had to alter the way the call is dealt with and reject the excess traffic.

Most on this forum have our CAD so the point is moot but FYI, we've got a lot of Home Assistant users of MQTT. We tell people that MQTT isn't meant for half hourly data but they are using brokers that seem to make a lot of redundant calls.

We hope to get our usual more frequent updates of the DCC data in place soon but wanted to explain why we are currently on overnight only. Please bear with us and check back in a day or two. Thank you.

Checking my pi-hole, since yesterday I can see 11676 DNS queries (in less than 24 hours) which seems a bit excessive. Is there anything we can do with regards to that? Just asking as they are throttling the calls and I am not (and probably others) able to get info at all to HA.

smithbill17 commented 2 years ago

Since 1st Feb, I seem to be only getting a single reading at 5-6am every 24hrs (which is obviously pretty useless). If we're still making 1000s of DNS queries every day, then surely the only way forward is to stop retrieving 6 data points every 2mins and instead retrieve the same data every 30mins?

si458 commented 2 years ago

@smithbill17 i just suggested this in another issue https://github.com/HandyHat/ha-hildebrandglow-dcc/issues/126#issuecomment-1032600786

xhemp commented 2 years ago

@smithbill17 i just suggested this in another issue https://github.com/HandyHat/ha-hildebrandglow-dcc/issues/126#issuecomment-1032600786

Nice, I haven't noticed that issue. I've seen a scan_interval on sensor.py, I changed from 2 minutes to 10 to test and it seems to work.

si458 commented 2 years ago

@smithbill17 i just suggested this in another issue #126 (comment)

Nice, I haven't noticed that issue. I've seen a scan_interval on glow.py, I changed from 2 minutes to 10 to test and it seems to work.

my fix should help the issue #135

HandyHat commented 2 years ago

Hey all, thanks for your work here. I agree - we are currently making too many calls to the Hildebrand API. I have got some suggestions on how to reduce this impact, and would love your feedback in the discussion: https://github.com/HandyHat/ha-hildebrandglow-dcc/discussions/191

peter-dolkens commented 2 years ago

Just to comment - the DNS queries are highly unlikely to be going anywhere near Hildebrand. Your API requests is a different discussion, which you already have underway, but the DNS requests should be cached in numerous places:

The local system - most OSes will have some level of DNS cache
The local router - most routers will set themselves as the DNS server, and forward requests to your configured DNS providers (or the ISP configured providers) - allowing for local network name resolution.
Your upstream DNS servers - I can pretty much guarantee that your DNS queries are unlikely to be making it past your ISP / Cloudflare / Google / OpenDNS servers. Those servers will almost definitely be caching for the configured TTL. if not longer.
Cascading DNS servers across the internet - If you somehow do manage to get past your configured upstream, then their configured upstream servers are also likely to be caching records.
The domain registrar - It's unlikely that Hildebrand hosts its own nameservers. Far more likely is that they've given this job to Cloudflare, or Fastly, or Godaddy, or some other big name that is designed to serve immense amounts of DNS.

So we should definitely be careful to make sure that we're not making excessive DNS queries, but rest assured we're not DDoSing Hildebrand with this integration :)

HandyHat commented 1 year ago

v1.0.0 of the integration implements many of the ideas in #191 and should reduce the number of DNS lookups and the load on Hildebrand's servers. If you are still seeing excessive lookups, please say!

HandyHat / ha-hildebrandglow-dcc

1000s of DNS lookups to api.glowmarkt.com #55