search

pi-hole / FTL

The Pi-hole FTL engine
https://pi-hole.net
Other
1.34k stars 187 forks source link

Client ipv6 hostnames is not resolving and not being stored in FTL database #1978

Open moya2162 opened 1 month ago

moya2162 commented 1 month ago

Problem:

Pihole does assign hostname to ipv6 clients addresses properly when resolver.networkNames=true. When examining pihole-FTL.db only Pihole correctly assigns MAC address values to ipv4 and ipv6 client addresses, however hostname is left blank.

When Pihole identifies an ipv6 address it adds it to the database without resolving the hostname. When the same clients ipv4 address is discovered and resolved, Pihole then is able to correlate MAC address of the same device and understand it is the same device, however because the ipv6 address was discovered first, according to the logs its not new and is skipped.

Build:

arm64
docker
Core vDev (development-v6, 7b19b650)
FTL vDev (fix/resolver, 090c1ada)
Web interface vDev (development-v6, 24977e86)

Logs:

Here are some logs of when it does work. I ran a simple test on a single client. I first disabled ipv6, the client hostname was established through ipv4. I then enabled ipv6, and then the ipv6 hostname of that was established, however not on the first attempt:

IPv4

2024-05-28 21:07:08.128 DEBUG_RESOLVER Trying to obtain host name of "192.168.2.30" from network_addresses table
2024-05-28 21:07:08.130 DEBUG_RESOLVER  ---> not found
2024-05-28 21:07:08.579 DEBUG_RESOLVER Trying to resolve 192.168.2.30
2024-05-28 21:07:08.579 DEBUG_RESOLVER Resolving PTR "30.2.168.192.in-addr.arpa" on 127.0.0.1#53 (TCP)
2024-05-28 21:07:08.586 DEBUG_RESOLVER DNS query for PTR "30.2.168.192.in-addr.arpa" returned status NoError (0)
2024-05-28 21:07:08.586 DEBUG_RESOLVER Answer 0 is PTR "30.2.168.192.in-addr.arpa" => "XXXXX.lan"
2024-05-28 21:07:08.586 DEBUG_RESOLVER Client 192.168.2.30 -> "XXXXX.lan" is new

IPv6

2024-05-28 21:08:22.857 DEBUG_RESOLVER Trying to obtain host name of "fd71:2002:f23:0:f50c:11f0:e53b:8d1c" from network_addresses table
2024-05-28 21:08:22.859 DEBUG_RESOLVER  ---> not found
2024-05-28 21:08:22.861 DEBUG_RESOLVER getIfaceFromIP(): "SELECT interface FROM network JOIN network_addresses ON network_addresses.network_id = network.id WHERE network_addresses.ip = ? AND interface != 'N/A' AND interface IS NOT NULL;" with ? = "fd71:2002:f23:0:f50c:11f0:e53b:8d1c"
2024-05-28 21:08:23.704 DEBUG_RESOLVER Trying to resolve fd71:2002:f23:0:f50c:11f0:e53b:8d1c
2024-05-28 21:08:23.705 DEBUG_RESOLVER Resolving PTR "c.1.d.8.b.3.5.e.0.f.1.1.c.0.5.f.0.0.0.0.3.2.f.0.2.0.0.2.1.7.d.f.ip6.arpa" on 127.0.0.1#53 (TCP)
2024-05-28 21:08:23.734 DEBUG_RESOLVER DNS query for PTR "c.1.d.8.b.3.5.e.0.f.1.1.c.0.5.f.0.0.0.0.3.2.f.0.2.0.0.2.1.7.d.f.ip6.arpa" returned status Refused (Query Refused) (5)
2024-05-28 21:08:23.734 DEBUG_RESOLVER Trying to obtain host name of "fd71:2002:f23:0:f50c:11f0:e53b:8d1c" from network_addresses table
2024-05-28 21:08:23.736 DEBUG_RESOLVER  ---> not found
2024-05-28 21:08:23.737 DEBUG_RESOLVER Client fd71:2002:f23:0:f50c:11f0:e53b:8d1c -> "" is new
2024-05-28 21:08:55.773 DEBUG_RESOLVER Skipping client fd71:2002:f23:0:f50c:11f0:e53b:8d1c -> "" because it is not new
2024-05-28 21:09:34.501 DEBUG_RESOLVER Trying to obtain host name of "fd71:2002:f23:0:f50c:11f0:e53b:8d1c" from network_addresses table
2024-05-28 21:09:34.502 DEBUG_RESOLVER Found database host name (same device) fd71:2002:f23:0:f50c:11f0:e53b:8d1c -> XXXXX.lan

Here is logs of it not working:

2024-05-28 20:24:43.691 DEBUG_RESOLVER Trying to resolve 2601:204:0:cf3c:d4b7:ce55:23bd:8da4
2024-05-28 20:24:43.692 DEBUG_RESOLVER Resolving PTR "4.a.d.8.d.b.3.2.5.5.e.c.7.b.4.d.c.3.f.c.0.0.0.0.4.0.2.0.1.0.6.2.ip6.arpa" on 127.0.0.1#53 (TCP)
2024-05-28 20:24:43.702 DEBUG_RESOLVER DNS query for PTR "4.a.d.8.d.b.3.2.5.5.e.c.7.b.4.d.c.3.f.c.0.0.0.0.4.0.2.0.1.0.6.2.ip6.arpa" returned status Refused (Query Refused) (5)
2024-05-28 20:24:43.702 DEBUG_RESOLVER Trying to obtain host name of "2601:204:0:cf3c:d4b7:ce55:23bd:8da4" from network_addresses table
2024-05-28 20:24:43.704 DEBUG_RESOLVER  ---> not found
2024-05-28 20:24:43.705 DEBUG_RESOLVER Client 2601:204:0:cf3c:d4b7:ce55:23bd:8da4 -> "" is new
...
...
2024-05-28 20:34:45.446 DEBUG_RESOLVER Skipping client fd71:2002:f23:0:d4b7:ce55:23bd:8da4 -> "" because it is not new

Notice that most of the logs between working and not working are the same with the only difference being the last few lines. Not entirely sure as to what promts pihole to try to obtain host name from network_addresses table in the logs where it works vs where it doesnt work.

Additional Info:

Another problem is that pihole-FTL.db is not updating ipv6 hostname after it is found.

Network Tab

Screenshot 2024-05-28 212455

Database

Screenshot 2024-05-28 212804

And i believe another byproduct of this this issue problem is the following issue: https://github.com/pi-hole/pi-hole/issues/5544

DL6ER commented 1 month ago

The one occurrence you have seen

2024-05-28 21:09:34.501 DEBUG_RESOLVER Trying to obtain host name of "fd71:2002:f23:0:f50c:11f0:e53b:8d1c" from network_addresses table
2024-05-28 21:09:34.502 DEBUG_RESOLVER Found database host name (same device) fd71:2002:f23:0:f50c:11f0:e53b:8d1c -> XXXXX.lan

likely comes from an API call which may have tried to resolve the IPv6 host name on its own for the API response.

Yes, "the order matters" is the problem here. Once looking up has failed once, IPv6 addresses are not tried again unless you configured resolver.refreshNames=ALL. The issue is that we cannot know that we learned a host name through another IP address of the same device by now. At any later point (e.g. after a restart of pihole-FTL), Pi-hole should be able to resolve this address through the "same device" trick.

You second example with the IPv6 GUA address does not work because this device seemingly is not known also not through another device? I'm somewhat lacking a it of context here. (is it the same device? if so, we could try to identify why Pi-hole wasn't able to see the MAC address)

So the question is: What could be the compromise here? Seemingly simply skipping because of "not new" is an issue here. But I also wouldn't want to simply do the database lookup always (and just skip the PTR) for IPv6 addresses because all this skipping was introduced because users complaining about poor performance in networks with many devices rotating their IPv6 address in matters of seconds, i.e., they had several thousand IPv6 "clients" during a normal day. Performing multi-stage database lookups for each of them every minute is definitely not the way to go.

moya2162 commented 4 weeks ago

Thank you for providing some context here. In all examples above the devices logs i posted are for the same client. I think most of my problems are from a lack of understand of how Pihole works internally. I try my best with flipping switches and observing outcomes to try to learn. Thank you for working with me, being patient, and educating me.

I have taken some time to play around with the settings and try to figure what triggers Pihole to write hostnames to pihole-FTL.db. As you stated above I played with setting resolver.refreshNames=ALL to see if having Pihole trigger a PTR request for IPv6 would cause resolving and writing IPv6 client hostnames into the database via the "same device" mechanism. So far based on my testing I think it is doing it, but there seems to be a delay. Does Pihole store recently defined hostnames in a cache for a while and then write updates to pihole-FTL.db?

DL6ER commented 4 weeks ago

here seems to be a delay. Does Pihole store recently defined hostnames in a cache for a while and then write updates to pihole-FTL.db?

Yes, it is buried down in the fine details as we have to balance I/O on the database (Pi-hole is still designed to run on Raspberry Pi like hardware and SD card I/O is heavily wear-limited). The delay shouldn't bee all that long, though. It is user-controlled via the config setting database.DBinterval defaulting to 60 seconds.

I'm still unsure about how the compromise we are looking for should look like exactly....

moya2162 commented 4 weeks ago

Could the compromise be by making a new setting which we could call resolver.retryoveride= with options as true or false. If set to true, every time an ip comes back as "not new" it can retry for a name rather than skipping. If set to false, it stays as currently programed?

Or another way could be to instead of going with true or false, a number is the value provided, in seconds, where a user could override the refresh of "not new". Then when a "not new" address shown up and lastQuery > resolver.retryoveride then a check is done. If no value is provided it stays as currently programmed.

Not sure if these ideas have performance implications.