Weird DNS requests via PiHole

[ArchaicRider]

I'm running Pihole on a Pi Zero and am gradually pointing all my devices to use this as a DNS filter.

I've just pointed my motioneye OS (on a Pi 2) using the static IP configuration and now PiHole is recording entries such as

2021-10-26 16:49:36 AAAA no-cache" 192.168.0.204 and 2021-10-26 16:49:36 A no-cache" 192.168.0.204

Update: Just to add that this pairing occurs once every hour.

Both the no-cache and the double quote after it make me feel it's a coding issue but I don't know where to start.

My static_ip.conf is

STATIC_IP="192.168.0.204/24"
STATIC_GW="192.168.0.1"
STATIC_DNS="192.168.0.18"

Both the machines are working fine, I just don't like inconsistencies like this!

Where should I start looking to see why this is happening?

Cheers

Archaic Rider

[starbasessd]

MotionEye or motionEyeOS? If motionEyeOS which version? Anything in the log for Pihole as to the actual DNS request from motionEye? If you don't use names for anything that the Pi2 uses, set DNS to 127.0.0.1

[starbasessd]

The only thing I can think of using DNS would be Time Server (like pool.ntp.org) If you have an Internal to your network NTP or Web server, use that for Time server

[ArchaicRider]

Thanks @starbasessd

This is for MotionEyeOS, from the admin panel...

motionEye Version | 0.42.1
Motion Version | 4.2.2+gitUNKNOWN
OS Version | motionEyeOS 20200606

The log entry in PiHole is as I put it above, other requests from the same device look like...

2021-10-26 17:49:37 | AAAA | google.com | 192.168.0.204

So it appears that the device is asking for the DNS entry for no-cache"

Don't have internal servers (yet) so can't use that and don't want to point the DNS to 127.0.0.1 in case that breaks something else e.g. email sending or web-based updates!

[starbasessd]

no-cache is an instruction, basically (very basically) I need the current address, and not your cached entry. I don't have that issue in production (could test on a quick and dirty). Let me know if you are very concerned. If you are using email messaging, and so forth, then no, moving the DNS server to localhost would be 'bad'.

[ArchaicRider]

Thanks for the background on no-cache

I'm not particularly concerned about it, everything appears to be working.

I was just wondering if it's a bad config on my part either for PiHole or MotionEyeOS

[ArchaicRider]

From pihole.log

Oct 26 17:42:48 dnsmasq[21169]: query[AAAA] smtp.gmail.com from 192.168.0.204
Oct 26 17:43:35 dnsmasq[21169]: query[A] smtp.gmail.com from 192.168.0.204
Oct 26 17:43:35 dnsmasq[21169]: query[AAAA] smtp.gmail.com from 192.168.0.204
Oct 26 17:49:37 dnsmasq[21169]: query[A] no-cache" from 192.168.0.204
Oct 26 17:49:37 dnsmasq[21169]: query[AAAA] no-cache" from 192.168.0.204
Oct 26 17:49:37 dnsmasq[21169]: query[A] google.com from 192.168.0.204
Oct 26 17:49:37 dnsmasq[21169]: query[AAAA] google.com from 192.168.0.204
Oct 26 18:00:00 dnsmasq[21169]: config 192.168.0.204 is NXDOMAIN
Oct 26 18:49:37 dnsmasq[21169]: query[A] no-cache" from 192.168.0.204
Oct 26 18:49:37 dnsmasq[21169]: query[AAAA] no-cache" from 192.168.0.204
Oct 26 18:49:37 dnsmasq[21169]: query[A] google.com from 192.168.0.204
Oct 26 18:49:37 dnsmasq[21169]: query[AAAA] google.com from 192.168.0.204

Perhaps an issue with PiHole not dealing with the no-cache" attribute?

[starbasessd]

Possible. I only run pihole occasionally (when grand kids or non-family visit). I have some stuff whitelisted, a lot of stuff blacklisted, and find too many 'issues' with it, personally. I run a lot of my own services, DNS, NTP (using GPS as source), static DHCP, etc. (Even had an upsidown internet proxy for a while due to some teenage hackers) along with honeypots and DMZ and such.

[starbasessd]

Practicing Google Foo, and found this: https://github.com/pi-hole/pi-hole/issues/1794 NXDOMAIN might be because of any request, like talking to a camera you have listed as 'front.local' or a NTP server listed with a bad spelling ( pole.ntp.org instead of pool.ntp.org ), or something along those lines... Lots of possibilities depending on features you are using...

[ArchaicRider]

I've read it and have to admit most of it goes above my head!

I'll keep trying though...

[ArchaicRider]

I've been looking at this again and I've got a time series of requests on PiHole from the MotionEyeOS device

2021-10-27 17:49:46 AAAA    google.com      192.168.0.204   OK, answered by dns.google#53   IP (24.1ms) 
2021-10-27 17:49:46 A   google.com      192.168.0.204   OK, answered by dns.google#53   IP (21.1ms) 
2021-10-27 17:49:46 AAAA    no-cache"       192.168.0.204   OK (cache)          NXDOMAIN (0.5ms)    
2021-10-27 17:49:46 A   no-cache"       192.168.0.204   OK (cache)          NXDOMAIN (0.6ms)    
2021-10-27 17:12:11 AAAA    smtp.gmail.com  192.168.0.204   OK (cache)              IP (0.3ms)  
2021-10-27 17:12:11 A   smtp.gmail.com  192.168.0.204   OK (cache)              IP (0.3ms)  
2021-10-27 16:49:46 AAAA    google.com      192.168.0.204   OK, answered by dns.google#53   IP (21.2ms) 
2021-10-27 16:49:46 A   google.com      192.168.0.204   OK, answered by dns.google#53   IP (20.4ms) 
2021-10-27 16:49:46 AAAA    no-cache"       192.168.0.204   OK (cache)          NXDOMAIN (0.5ms)    
2021-10-27 16:49:46 A   no-cache"       192.168.0.204   OK (cache)          NXDOMAIN (0.5ms)

I wondering if somewhere in the OS code there is a trigger once an hour to check that google.com exists as the weird no-cache" entry appears in the same second.

Not the same time as checking the smtp so doesn't appear to be related to that.

If I knew how to use github I'd try looking in the code!

[starbasessd]

If you have a spare Pi, or a VM, or something, you can install the motionEye code any of a number of different ways, and look at the python scripts. You can find the install instructions here: https://github.com/ccrisan/motioneye/wiki/Installation where you'll find out all the flavors of linux methods, and docker image. If you tell me what setup you have, and if you want to learn python, I'll try to get yo usome instructions.

[starbasessd]

In Pihole, if you go to Query Log, then click on the client column, then in the search box put in the IP address of the PiCam, it should tell you the domain it was searching for: 2021-10-27 06:51:31 | A | e1329.g.akamaiedge.net | 192.168.x.x | OK (cache) | N/A |   or 2021-10-27 15:11:18 | A | wpad.internal-domain.local | 192.168.x.y | OK, answered by 192.168.x.z#53 | NXDOMAIN (1.1ms) |   for example.

[ArchaicRider]

Yes, that's what the extract is above, it appears that the MotionEyeOS (*.204) is searching for no-cache" followed by google.com and it happens about once an hour. I don't know if it's an erroneous request from the MEOS or PiHole not dealing with a no-cache DNS request appropriately.

As the pihole is wireless I did wonder about seeing if I could listen to the wifi with something like WireShark(?) and see if that could show if if it's the request that's wrong.

Thank you for the offer of help. Don't have a spare Pi but I can setup a linux VM on my PC (won't be a Pi VM though which would be the best distribution?), happy to try to learn (or perhaps just understand) Python as I've dabbled in a number of languages over the past 30 years!

I've downloaded the code for MotionEye and MotionEye OS and a search through it for no-cache or google.com didn't lead me to anything that screamed "this is what you're looking for" :-)

[starbasessd]

In the motioneye code, all the python files are in the ./usr/local/lib/python2.7/dist-packages/motioneye folder tree if you install on a Linux (I prefer Debian Buster and Ubuntu 20.04 myself but there is the WSL2 on Win10 environment, too) If you use github and there are MANY W10 git apps available they are in the motioneye/motioneye folder tree. The only place I find 'no-cache' is in the handlers.py file: https://github.com/ccrisan/motioneye/blob/dev/motioneye/handlers.py but I can't tell you if that is 'causing' it...

[starbasessd]

Just added Debian 11 to WSL2, did an apt install git, then cloned git clone -b dev https://github.com/ccrisan/motioneye.git to do some occasional deep dives, as I don't want to waste time and effort to fire up a dev vm, especially if I am playing with docker or non-Pi environment on my desktop...

[zagrim]

The only place I find 'no-cache' is in the handlers.py file: https://github.com/ccrisan/motioneye/blob/dev/motioneye/handlers.py but I can't tell you if that is 'causing' it...

That's just setting HTTP request headers (to avoid streaming frames from a camera from getting cached) and I can't think of any reason those headers would leak to DNS queries.

[zagrim]

As the pihole is wireless I did wonder about seeing if I could listen to the wifi with something like WireShark(?) and see if that could show if if it's the request that's wrong.

@ArchaicRider I am not familiar with PiHole so I don't know if it supports installing extra applications), but it would be simpler to dump traffic there with command line tools like tshark or directly with dumpcap and the transfer the output file to a desktop computer for viewing in Wireshark. Since at least MEOS doesn't really support installing anything extra on it (according to my understanding) capturing traffic there is already out of the question.

One can also monitor wifi traffic from another box, but that requires having a wifi adapter that supports monitor mode (and perhaps some other extra things are required, too, I've never done that myself). This would be the only option if it is not feasible to install e.g. dumpcap on PiHole, but I can't give any detailed instructions for that since I lack that experience. There should be enough of tutorials for that in the Web, though.

Anyway, I think inspecting the network traffic would be the most straight-forward way of figuring this out, and that would probably be the option I'd attempt if I was in your situation.

[ArchaicRider]

Right,managed to install tshark on pihole and work out the filter for just traffic from my motioneye machine.

First pass around the hourly weird DNS request and I caught one!

62 2347.026243955 192.168.0.204 → 192.168.0.18 DNS 69 Standard query 0xf0ff A no-cache"
   63 2347.026583951 192.168.0.204 → 192.168.0.18 DNS 69 Standard query 0x209e AAAA no-cache"
   64 2347.028280930 192.168.0.18 → 192.168.0.204 DNS 69 Standard query response 0xf0ff No such name A no-cache"
   65 2347.029570915 192.168.0.18 → 192.168.0.204 DNS 69 Standard query response 0x209e No such name AAAA no-cache"
   66 2347.093542138 192.168.0.204 → 192.168.0.18 DNS 70 Standard query 0x9b9c A google.com
   67 2347.093929133 192.168.0.204 → 192.168.0.18 DNS 70 Standard query 0x66d7 AAAA google.com
   68 2347.109779941 192.168.0.18 → 192.168.0.204 DNS 86 Standard query response 0x9b9c A google.com A 172.217.169.46

Reading that, it seems to be that the motioneye machine is definitely sending a DNS request for no-cache" for A & AAAA records.

Now I need to work out where in the motioneye code I'd look for that. Is it motioneye or perhaps the underlying OS?

Such fun!

[starbasessd]

I have no idea, as stated before, the only place I can find "no-cache" anywhere in motioneye code is in handlers.py. Weird.

[zagrim]

I couldn't find any explanation for this by grepping MEOS sources, either (a lot of things match with "cache" but they seemed to be build related - although there were so many matches that I didn't spend much thought on each single one). Anyway, I think this could be caused by some dependency of ME, or MEOS more specifically maybe. To research that we'd need to have at least some idea of the context for that DNS query. I mean, something is clearly wanting to have up-to-date IP address for google.com so the question is for what purpose? Is it just for doing network connectivity checking (being sure to use fresh IP to not trigger false alarms in some edge cases), or for using some of the Google services? Google APIs are under googleapis.com and authentication to Google uses accounts.google.com so that would suggest it's not related to Google API usage (like the Drive upload feature).

[ArchaicRider]

I'm now looking at this a different way, that series of DNS calls to pihole occurs every hour, therefore something is configured to happen every hour.

I've been checking through the conf files in /data/etc and found an interesting file, date.conf

DATE_METHOD=http
DATE_HOST=google.com
DATE_NTP_SERVER=
DATE_TIMEOUT=10
DATE_INTERVAL=3600

It has the domain that is queried after the no-cache" request, an interval of 3600, which in seconds is an hour, and a blank entry for DATE_NTP_SERVER=. Could this be the cause?

Does this config tell the device to check the date & time once an hour using google.com for the date and nothing for the time?

Does the code put no-cache on the NTP DNS request for this check to ensure it's got "live" data?

Is the code then putting a blank string in that DNS request for the NTP server which then makes no-cache look like the domain it's requesting?

I suppose a way to check would be to put something in the NTP server entry, not certain what though, or how! Can I just edit date.conf or do I need to do it via the web-interface?

[ArchaicRider]

Ok, I've found the cause and remedy, not a fix though.

It's down to the Date Method in Expert Setting in the web interface

If I choose Date Method of HTTP with Date HTTP Host of google.com then on reboot, pihole is asked for the following DNS lookups

2021-11-07 15:39:27 | AAAA | google.com | 192.168.0.204 | OK, answered by dns.google#53 | IP (27.7ms)
2021-11-07 15:39:27 | A | google.com | 192.168.0.204 | OK, answered by dns.google#53 | IP (23.4ms)
2021-11-07 15:39:27 | AAAA | no-cache" | 192.168.0.204 | OK (cache) | NXDOMAIN (0.2ms)
2021-11-07 15:39:27 | A | no-cache" | 192.168.0.204 | OK (cache)

If I change to NTP with uk.pool.ntp.org then the following are requested

2021-11-07 15:37:17 | AAAA | uk.pool.ntp.org | 192.168.0.204 | OK (cache) | NODATA (0.9ms)
2021-11-07 15:37:17 | A | uk.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53 | IP (24.3ms)
2021-11-07 15:37:09 | AAAA | 2.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53 | IP (32.8ms)
2021-11-07 15:37:09 | A | 2.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53 | IP (17.3ms)
2021-11-07 15:37:09 | AAAA | 1.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53 | NODATA (25.3ms)
2021-11-07 15:37:09 | A | 1.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53 | IP (16.8ms)
2021-11-07 15:37:09 | AAAA | 0.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53 | NODATA (72.1ms)
2021-11-07 15:37:09 | A | 0.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53

If I change to SNTP (recommended) then I get

2021-11-07 15:41:17 | AAAA | uk.pool.ntp.org | 192.168.0.204 | OK (cache) | NODATA (1.2ms)
2021-11-07 15:41:17 | A | uk.pool.ntp.org | 192.168.0.204 | OK (cache) | IP (1.2ms)
2021-11-07 15:41:17 | AAAA | uk.pool.ntp.org | 192.168.0.204 | OK (cache) | NODATA (1.3ms)
2021-11-07 15:41:17 | A | uk.pool.ntp.org | 192.168.0.204 | OK, answered by dns.google#53

So my fix / work-around is to use the recommended SNTP and the no-cache" DNS requests cease.

Don't understand why the same two requests are duplicated on HTTP and SNTP and I guess the multiple n.pool.ntp.org requests are something to do with the NTP protoco.

For now I'm happy to understand and stop the no-cache requests from occurring! :-)

Thanks to all for helping me move forward with this.

[zagrim]

Nice work!

I guess the no-cache" queries might be due to some quoting issue in set_current_date_http here https://github.com/ccrisan/motioneyeos/blob/dev/board/common/overlay/etc/init.d/S50date#L36 (what is intended to be HTTP header content ends up interpreted otherwise by the shell), but I can't be sure of that. Anyway, that would be MEOS-only issue, not involving ME.

[ArchaicRider]

Found time to look at this again, and it's interesting.

Looking at the code to which @zagrim has pointed, I've deconstructed it to get the following curl instruction

curl -v -s -m 10 -H \"Cache-Control: no-cache\" -X "GET" http://www.google.com

I've three pis PiZero V1.1 - running pihole, Pi 3 Model B+ - used as an FTP server, and Raspberry Pi Model B Rev 2 - MotionEyeOS. All have different versions of curl and all give the error somewhere in the output of Could not resolve host: no-cache". To be expected really as the string contains the escape sequence of \".

Running the instruction

curl -v -s -m 10 -H "Cache-Control: no-cache" -X GET http://www.google.com

i.e. without the two \, gives no errors at all.

Makes me wonder if the escaping mechanism of \" is not working, I've tested the command

curl -v -s -m 10 -H 'Cache-Control: no-cache' -X GET http://www.google.com and it appears to work.

Therefore I'm wondering if changing L36 in the file referenced by @zagrim from

curl_args="-v -s -m ${DATE_TIMEOUT} -H \"Cache-Control: no-cache\" -X GET"

to

curl_args="-v -s -m ${DATE_TIMEOUT} -H 'Cache-Control: no-cache' -X GET"

would solve this, admittedly very small, bug?

Can I just go in and amend that file on my MotionEyeOS installation and reboot?

Cheers

AR

[starbasessd]

Not likely, as the python scripts are compiled (.pyc) and would have to be de-compiled, edited and re-compiled, but to get to them, you'd have to remount the / partition (partition2) as read-write to begin the whole operation, unless you want to try part 1 of the above on another linux machine that can edit the partitions directly...

[starbasessd]

Let us know how it goes...

[zagrim]

@ArchaicRider You are on the point there. Quoting is sometimes tough in shell scripts... I don't quite see why that even fails like that but maybe that's some special case between parameter expansion and command substitution.

Since that file is a shell script that is installed in /etc/init.d, I think patching that file in your installation should not be quite as hard as @starbasessd put it above (no need to worry about (de)compiling Python code at any stage), although I'm not at all familiar with the MEOS setup on that level.

[ArchaicRider]

Fixed it!

Played around for ages with the curl_args line with different combinations of escape characters and string terminators but with no joy.

Discovered that probably didn't need both the -v & -s switches for curl so removed the -s.

Eventually moved all the curl arguments into the curl line itself and the no-cache" error disappeared.

Therefore deleting line 36 and amending line 38 (numbered as before the deletion) resolves the issue.

Possibly the code is less readable but it's not that bad.

In over 30 years of using OSS this is the first time I've been able to contribute so now I've got to work out how to submit a change!

And, after all this, I'm going to change my configuration back to using SNTP to set the time as that's the recommended setting but at least I'll know the HTTP way does not have that bug!

Thanks to all for your help

AR