Cloudflared stops resolving DNS.

ghost commented 6 years ago

I am not sure how to diagnose the issue but hoping someone can help. Not overly sure if it is an issue with Clouflared.

I have successfully setup Cloudflared to act as a DNS server and using it with Pi-Hole. I have manually specified my DNS on a laptop and that works perfectly.

Unfortunately is I change my DNS in the router Cloudflared stops resolving DNS. I have double checked this by connecting using SSH and manually attempting a DNS query and nothing is returned. It stopped working immediately after changing the router to hand out the DNS server.

The steps taken to setup closely follow: https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/

hasmar04 commented 4 years ago

My fix before I switched to unbound was to use these unofficial builds: https://hobin.ca/cloudflared/releases/ All that is different is they are compiled using libraries that support the Pi Zero.

jankais3r commented 4 years ago

My fix before I switched to unbound was to use these unofficial builds: https://hobin.ca/cloudflared/releases/ All that is different is they are compiled using libraries that support the Pi Zero.

Those builds fix the segfault, but this not issue.

hasmar04 commented 4 years ago

My fix before I switched to unbound was to use these unofficial builds: https://hobin.ca/cloudflared/releases/ All that is different is they are compiled using libraries that support the Pi Zero.

Those builds fix the segfault, but this not issue.

My bad. Must be a different issue.

davideaicardi commented 4 years ago

Thank you guys, I'll try with dnscrypt

unbaiat commented 4 years ago

use https://github.com/pi-hole/pi-hole/wiki/DNSCrypt-2.0. latest release of argo is still not working

p-doyle commented 4 years ago

Even Cloudflare recommends using DNSCrypt... https://blog.cloudflare.com/deploying-gateway-using-a-raspberry-pi-dns-over-https-and-pi-hole/ I guess they've given up on cloudflared???

JustinFreid commented 4 years ago

I've been struggling with Cloudflared and I can see why they're recommending DNSCrypt-proxy: https://community.cloudflare.com/t/cloudflare-gateway-and-cloudflared/164083/23

acmacalister commented 4 years ago

Howdy y'all. We have been working on some updates to the DoH client for cloudflared, as well as some general usability improvements overall. I personally have been running the latest of cloudflared (2020.4.0) on my work Mac without any issues for the past couple weeks. From reading through the thread is this issue limited to the Pi?

I would love some steps to reproduce so we could get to the bottom of this issue.

JustinFreid commented 4 years ago

As I said, I switched to DNSCrypt-proxy (which I thought crashed only once but it turned out Cloudflare DNS itself had an issue) and the other person on the thread I linked to switched to using NextDNS. I would setup Pi Hole using the standard Raspberry Pi image with GUI, test out Pi Hole with Cloudflare DNS directly, then try switching to using Cloudflared for DoH and I bet it will fail after half a day.

nathang21 commented 4 years ago

FWIW I’ve been running it for almost 6 months without issue. Rocker Pihole

cloudfared using the Pi 3B+. Here’s my repo, I restart and update weekly via Cron not sure if that would mitigate this issue.

https://gitlab.com/nathang21/nate-and-xtina-home/-/tree/master/pi-hole

On Fri, May 1, 2020 at 2:34 PM Justin Freid notifications@github.com wrote:

As I said, I switched to DNSCrypt-proxy (which I thought crashed only once but it turned out Cloudflare DNS itself had an issue) and the other person on the thread I linked to switched to using NextDNS. I would setup Pi Hole using the standard Raspberry Pi image with GUI, test out Pi Hole with Cloudflare DNS directly, then try switching to using Cloudflared for DoH and I bet it will fail after half a day.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cloudflare/cloudflared/issues/23#issuecomment-622577736, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4CUJXTYD6GQVTK25CJKNDRPM56TANCNFSM4E4IITWQ .

davideaicardi commented 4 years ago

Howdy y'all. We have been working on some updates to the DoH client for cloudflared, as well as some general usability improvements overall. I personally have been running the latest of cloudflared (2020.4.0) on my work Mac without any issues for the past couple weeks. From reading through the thread is this issue limited to the Pi?

I would love some steps to reproduce so we could get to the bottom of this issue.

I personally followed cloudflare’s guide and nothing worked, on pi not at all, I did it a month ago I don’t remember, and my Mac I tried again but resolving took many second every request. I’m waiting for my Mikrotik so I can get doh network wide.

p-doyle commented 4 years ago

I am still using cloudflared. I had to write a script to monitor the log output from the cloudflared service and then restart it if it started throwing errors.

To replicate for me all that needs to happen is to lose internet access, for example, if I restart my router. I am using Rasbian Buster on an RPI 3B+. After losing internet, and with DNS requests still coming in, it starts throwing errors like this:

May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query:
May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query:
May 01 22:48:30 pihole cloudflared[31738]: time="2020-05-01T22:48:30+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query:

It continues to do this until the service is restarted.

If there is additional info you need let me know.

JustinFreid commented 4 years ago

Sorry, I should also say that the other person in the thread I linked to and I were using Cloudflare Gateway with the personalized DoH query URL.

On Fri, May 1, 2020 at 17:53 p-doyle notifications@github.com wrote:

I am still using cloudflared. I had to write a script to monitor the log output from the cloudflared service and then restart it if it started throwing errors.

To replicate for me all that needs to happen is to lose internet access, for example, if I restart my router. I am using Rasbian Buster on an RPI 3B+. After losing internet, and with DNS requests still coming in, it starts throwing errors like this:

May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: May 01 22:48:30 pihole cloudflared[31738]: time="2020-05-01T22:48:30+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query:

It continues to do this until the service is restarted.

If there is additional info you need let me know.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cloudflare/cloudflared/issues/23#issuecomment-622583263, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADJPKPDKHPJXTN24TSWGUDRPNAHJANCNFSM4E4IITWQ .

-- --Justin Freid (sent from mobile) +1 917.720.6218 http://justinfreid.com

ratsputin commented 4 years ago

I also had this same issue whenever I would take an Internet hit. It was especially frustrating as I have automatic failover to a secondary provider. To addres this, I made the migration to dnscrypt-proxy but in the process had to make a change to my configuration to get it up and running. I'm wondering if this might be the root cause behind cloudflared losing its mind.

In short, I have my edge router set to do a u-turn NAT such that any DNS lookups targeted for the Internet are redirected back to the Pi-Hole server where cloudflared is running. If, when internet connectivity goes down, cloudflared attempts to do a DNS lookup and gets into some sort of loop, that might explain this behavior.

In my case, I had to put an exception in for the NAT rule to allow the Pi-Hole server to access the Internet for DNS to get dnscrypt-proxy working.

At this point, I don't have any interest in pursuing cloudflared any further as dnscrypt-proxy is working fine for me but I'm curious if those who have this issue might be in a similar setup. Especially @p-doyle .

p-doyle commented 4 years ago

@ratsputin I have a similar setup. pfsense router with a NAT rule to redirect port 53 to my pi-hole. Cloudflared shouldn't be using port 53 for any dns lookups though, right?

war59312 commented 4 years ago

Same issue here and seems no one at cloudflare cares.

Switch over to dnscrypt-proxy as well. Screw it.

TownLake commented 4 years ago

Hi from Cloudflare. Like @acmacalister mentioned, we've made some improvements to packaging and DoH issues. That said, we still haven't been able to track down the cause of the issues with Raspberry Pi devices. I know it's frustrating. We have the PiHole DoH issues on our list, but don't have an ETA yet when you could expect a fix. We'll update this thread when we do. Thanks for the patience and feedback.

extric99 commented 3 years ago

Any progress on this one. My subjective feeling is that the package has gotten more stable. I still see the errors in the log but it became rare that i would lose all connectivity and need to restart the service.

bxlouis commented 3 years ago

I am still using cloudflared. I had to write a script to monitor the log output from the cloudflared service and then restart it if it started throwing errors.

To replicate for me all that needs to happen is to lose internet access, for example, if I restart my router. I am using Rasbian Buster on an RPI 3B+. After losing internet, and with DNS requests still coming in, it starts throwing errors like this:
May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query:
May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query:
May 01 22:48:30 pihole cloudflared[31738]: time="2020-05-01T22:48:30+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query:
It continues to do this until the service is restarted.

If there is additional info you need let me know.

@p-doyle Could you please provide the script you used as I’d like to try this on my Rpi? Thanks!

FarawayMagnet commented 3 years ago

Hi from Cloudflare. Like @acmacalister mentioned, we've made some improvements to packaging and DoH issues. That said, we still haven't been able to track down the cause of the issues with Raspberry Pi devices. I know it's frustrating. We have the PiHole DoH issues on our list, but don't have an ETA yet when you could expect a fix. We'll update this thread when we do. Thanks for the patience and feedback.

hello, @acmacalister and @TownLake -- to add some extra info, this is not specific to rpi/arm devices. i'm running pihole + cloudflared on an amd64 ubuntu 20.04 virtual machine, and i'm experiencing the same issues others have noted here, i.e., losing wan connectivity via either my modem or firewall causes all upstream resolution to fail for between ten and twenty minutes, around which time it finally starts working again.

bxlouis commented 3 years ago

If this still interests someone, I implemented a workaround using Monit, which monitors cloudflared and restarts it as soon as it starts using too much CPU ressource. It is definitely not clean, but it works.

check process cloudflared matching "cloudflared" start program = "/bin/systemctl start cloudflared" stop program = "/bin/systemctl stop cloudflared" if cpu usage > 20% for 1 cycles then restart

That being said, I switched over to dnscrypt-proxy today. Super easy to configure, thoroughly documented with way more options and seems reliable and well maintained.

One cannot rely on a software with such an issue not addressed in 2 years.

dvejmz commented 3 years ago

FYI I opened a PR (#290) a while ago that works around this problem by allowing people to launch cloudflared proxy-dns with a parameter to limit the number of concurrent TCP connections to Cloudflare, which is the main cause of the runaway CPU usage issue. I've been running a version of cloudflared with this patch on my RPi 1 and it's been working very well so far.

netstx commented 3 years ago

I have the same issue (cloudflared stops responding to queries from PiHole once my internet "drops", i.e. rebooted my pfSense firewall).

My setup is a bit different: LXC container (on Proxmox) running latest Debian 10. PiHole is installed and using cloudflared as the dns proxy. All my DHCP clients use PiHole for DNS.

Prior to my last reboot, my router was up for 6+ months so I didn't catch this problem until now. I don't recall if this was an issue in earlier versions.

@acmacalister @TownLake I don't think this issue is unique to Raspberry Pi's, which I think I read above in the thread some people saying that.

EDIT: a few more details. Here is my apt source config on Debian:

deb http://pkg.cloudflare.com/ buster main

And here is a snippet of the logs once the internet disconnects:

Feb 26 21:36:59 pihole cloudflared[118]: 2021-02-27T03:36:59Z ERR failed to connect to an HTTPS backend "https://1.1.1.1/dns-query" error="failed to perform an HTTPS request: Post \"https://1.1.1.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Syslog for cloudflared starting:

Feb 26 21:45:47 pihole cloudflared[118]: 2021-02-27T03:45:47Z INF Initiating graceful shutdown due to signal terminated ...
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF Version 2021.2.5
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF Settings: map[config:/etc/cloudflared/config.yml no-autoupdate:true origincert:/etc/cloudflared/cert.pem proxy-dns:true proxy-dns-port:5053]
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF Adding DNS upstream url=https://1.1.1.1/dns-query
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF Adding DNS upstream url=https://1.0.0.1/dns-query
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF Starting DNS over HTTPS proxy server address=dns://localhost:5053
Feb 26 21:47:29 pihole cloudflared[115]: 2021-02-27T03:47:29Z INF cloudflared will not automatically update if installed by a package manager.

EDIT2: In my case I did not see increased CPU usage of my container during this problem. I tried to restart cloudflared with "systemctl restrart cloudflared" and it was taking a while to show as stopped, so I just shutdown the container and started it again. That solved the issue immediately (I am assuming it's because cloudflared restarted along with everything else).

martin3000 commented 3 years ago

Same here: failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request, connection reset by peer

p-doyle commented 3 years ago

I am still using cloudflared. I had to write a script to monitor the log output from the cloudflared service and then restart it if it started throwing errors. To replicate for me all that needs to happen is to lose internet access, for example, if I restart my router. I am using Rasbian Buster on an RPI 3B+. After losing internet, and with DNS requests still coming in, it starts throwing errors like this:
May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query:
May 01 22:48:29 pihole cloudflared[31738]: time="2020-05-01T22:48:29+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query:
May 01 22:48:30 pihole cloudflared[31738]: time="2020-05-01T22:48:30+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query:
It continues to do this until the service is restarted. If there is additional info you need let me know.
@p-doyle Could you please provide the script you used as I’d like to try this on my Rpi? Thanks!

It just uses the cysystemd python package to read the journal messages from the cloudflared service. It was sort of flaky though and probably wasn't the best way to do that. I have since switched to dnscrypt and haven't used the script in a while.

fonic commented 2 years ago

Hi from Cloudflare. Like @acmacalister mentioned, we've made some improvements to packaging and DoH issues. That said, we still haven't been able to track down the cause of the issues with Raspberry Pi devices. I know it's frustrating. We have the PiHole DoH issues on our list, but don't have an ETA yet when you could expect a fix. We'll update this thread when we do. Thanks for the patience and feedback.

It would seem that this has been resolved? At least for me, cloudflared now seems to operate much better on my Raspberry Pi 3 than when I lasted tested it about 9 month ago (back then, it was pretty much unusable). It still get a few cloudflared[428]: 2022-04-15T08:49:34Z ERR failed to connect to an HTTPS backend "https://9.9.9.9/dns-query" error="failed to perform an HTTPS request: Post \"https://9.9.9.9/dns-query\": dial tcp 9.9.9.9:443: connect: network is unreachable" errors per day, but resolving now seems to recover without having to restart cloudflared manually.

markpaul099 commented 2 years ago

Hi from Cloudflare. Like @acmacalister mentioned, we've made some improvements to packaging and DoH issues. That said, we still haven't been able to track down the cause of the issues with Raspberry Pi devices. I know it's frustrating. We have the PiHole DoH issues on our list, but don't have an ETA yet when you could expect a fix. We'll update this thread when we do. Thanks for the patience and feedback.

It would seem that this has been resolved? At least for me, cloudflared now seems to operate much better on my Raspberry Pi 3 than when I lasted tested it about 9 month ago (back then, it was pretty much unusable). It still get a few cloudflared[428]: 2022-04-15T08:49:34Z ERR failed to connect to an HTTPS backend "https://9.9.9.9/dns-query" error="failed to perform an HTTPS request: Post \"https://9.9.9.9/dns-query\": dial tcp 9.9.9.9:443: connect: network is unreachable" errors per day, but resolving now seems to recover without having to restart cloudflared manually.

I'm unlucky I guess, this still happens to me sometimes when when router restarts or lost connection I have to restart cloudflared manually

markpaul099 commented 2 years ago

I'm using this tutorial PiHole+DoH manual setting with cloudflare upstream

❯  cloudflared --version
cloudflared version 2022.5.0 (built 2022-05-03-0634 UTC)

KLEPTOROTH commented 2 years ago

Jesus Christ this has been a problem for FOUR YEARS..... guess I'm gonna jump ship to dnscrypt-proxy as well.

Very clear no one at cloudflare cares.

netstx commented 2 years ago

Jesus Christ this has been a problem for FOUR YEARS..... guess I'm gonna jump ship to dnscrypt-proxy as well.

Very clear no one at cloudflare cares.

I know, it's so strange...

p1r473 commented 2 years ago

Jesus Christ this has been a problem for FOUR YEARS..... guess I'm gonna jump ship to dnscrypt-proxy as well.

Very clear no one at cloudflare cares.

Use Unbound instead

fonic commented 2 years ago

Jesus Christ this has been a problem for FOUR YEARS..... guess I'm gonna jump ship to dnscrypt-proxy as well. Very clear no one at cloudflare cares.

Use Unbound instead

Unbound cannot act as a DOH client at the moment (*), thus it is not a suitable replacement for cloudflared.

(*) Pending feature request: https://github.com/NLnetLabs/unbound/issues/525

p1r473 commented 2 years ago

Jesus Christ this has been a problem for FOUR YEARS..... guess I'm gonna jump ship to dnscrypt-proxy as well. Very clear no one at cloudflare cares.

Use Unbound instead

Unbound cannot act as a DOH client at the moment (*), thus it is not a suitable replacement for cloudflared.

(*) Pending feature request: NLnetLabs/unbound#525

I have it successfully using DOT (DNS over TLS) which is great

#forward-ssl-upstream: yes - is the instruction to use DNS over TLS, in this case for all queries (name: “.”)
server:
    tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt
forward-zone:
    name: "."
    forward-tls-upstream: yes

    # Cloudflare DNS
    forward-addr: 1.1.1.1@853#cloudflare-dns.com
    forward-addr: 1.0.0.1@853#cloudflare-dns.com
    #forward-addr: 2606:4700:4700::1111@853#cloudflare-dns.com
    #forward-addr: 2606:4700:4700::1001@853#cloudflare-dns.com

    # NordVPN
    #forward-addr: 103.86.96.100@853#dns1.nordvpn.com
    #forward-addr: 103.86.99.100@853#dns2.nordvpn.com

    # Quad9
    #forward-addr: 2620:fe::fe@853#dns.quad9.net
    #forward-addr: 9.9.9.9@853#dns.quad9.net
    #forward-addr: 2620:fe::9@853#dns.quad9.net
    #forward-addr: 149.112.112.112@853#dns.quad9.net

fonic commented 2 years ago

I have it successfully using DOT (DNS over TLS) which is great

That is great, but still DOT, not DOH.

timhae commented 2 years ago

please let me know if I could provide more information to help resolve this issue

war59312 commented 2 years ago

This is the problem we have here and why still not fixed nearly 4.5 years later:

Kernighan’s law:

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

Cloudflare needs to hire a professional debugger, as clearly they have written too clever of code. ;)

fonic commented 2 years ago

Hi from Cloudflare. Like @acmacalister mentioned, we've made some improvements to packaging and DoH issues. That said, we still haven't been able to track down the cause of the issues with Raspberry Pi devices. I know it's frustrating. We have the PiHole DoH issues on our list, but don't have an ETA yet when you could expect a fix. We'll update this thread when we do. Thanks for the patience and feedback.

@TownLake ping

abelinkinbio commented 2 years ago

Hi All - I'd like to first apologize for the lack of clarity around this and other reported issues related to running cloudflared in proxy-dns mode. This is a feature we have not actively invested time and resources into for some time. Instead, we've pivoted our focus into making cloudflared a more robust and versatile connector of private resources. With that, we've begun to move more and more functionality into our WARP agent and are aiming to build a similar mode proxy-dns mode there in the future. That said, it's unlikely we'll pick up this ticket in the short term, but we do welcome any PRs for review and approval moving forward. Again, apologies for the silence here as I'm sure this has been a source of frustration. We'll be sure to update open and related tickets with a similar response moving forward.

timhae commented 2 years ago

There is a detailed guide for dnscrypt on raspberry pi here: https://blog.cloudflare.com/deploying-gateway-using-a-raspberry-pi-dns-over-https-and-pi-hole/

fonic commented 2 years ago

There is a detailed guide for dnscrypt on raspberry pi here: https://blog.cloudflare.com/deploying-gateway-using-a-raspberry-pi-dns-over-https-and-pi-hole/

There is also AdGuard Home which provides a nicely packaged all-in-one solution (one single executable) and supports DoT/DoH out of the box (both internally and externally).

cloudflare / cloudflared

Cloudflared stops resolving DNS. #23