cloudflare / cloudflared

Cloudflare Tunnel client (formerly Argo Tunnel)
https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide
Apache License 2.0
9.39k stars 836 forks source link

Cloudflared stops resolving DNS. #23

Open ghost opened 6 years ago

ghost commented 6 years ago

I am not sure how to diagnose the issue but hoping someone can help. Not overly sure if it is an issue with Clouflared.

I have successfully setup Cloudflared to act as a DNS server and using it with Pi-Hole. I have manually specified my DNS on a laptop and that works perfectly.

Unfortunately is I change my DNS in the router Cloudflared stops resolving DNS. I have double checked this by connecting using SSH and manually attempting a DNS query and nothing is returned. It stopped working immediately after changing the router to hand out the DNS server.

The steps taken to setup closely follow: https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/

yon2004 commented 6 years ago

Pull request #24 fixes this and gives more useful errors

andreagrandi commented 6 years ago

I also followed Scott instructions and the version I installed was cloudflared version 2018.4.8 (built 2018-04-26-1817 UTC). I had a similar problem but I can't understand if it's the same: at the beginning it was working fine, then everything got very slow and cloudflared was taking up to 95% of the CPU, slowing down even the PiHole dashboard. I tried a reboot, I tried to manually start it but nothing changed/improved. At the end I decided to deactivate it and switch back to a normal DNS.

Question: do you know how can I get a copy of the cloudflared version 2018.4.5 (built 2018-04-09-2155 UTC) binary, which is the same version used by Scott, just to understand if I'm getting the same problem with that version or if it could be a regression in the version I installed.

Cheers

andreagrandi commented 6 years ago

Just for reference, I started seeing this in the terminal (I cut a few lines to make it shorter):

pi@raspberrypi:~/argo-tunnel $ sudo ./cloudflared proxy-dns --port 54 --upstream https://1.1.1.1/.well-known/dns-query --upstream https://1.0.0.1/.well-known/dns-query
INFO[0000] Adding DNS upstream                           url="https://1.1.1.1/.well-known/dns-query"
INFO[0000] Adding DNS upstream                           url="https://1.0.0.1/.well-known/dns-query"
INFO[0000] Starting DNS over HTTPS proxy server          addr="dns://localhost:54"
INFO[0000] Starting metrics server                       addr="127.0.0.1:35905"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[1993] failed to connect to an HTTPS backend "https://1.1.1.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/.well-known/dns-query: read tcp 192.168.1.109:58962->1.1.1.1:443: read: connection timed out"
ERRO[2062] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2063] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2064] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2064] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
ERRO[2065] failed to connect to an HTTPS backend "https://1.0.0.1/.well-known/dns-query"  error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: EOF"
xetorixik commented 6 years ago

I do not use Scott's stuff but @andreagrandi got exactly the same problem. https://github.com/cloudflare/cloudflared/issues/23#issuecomment-387000967 Worked fine until yesterday.

Because there is no possibility to download the previous release, it is difficult to debug out where the problem is.

@andreagrandi If you can find the previous version. Please let me know.

andreagrandi commented 6 years ago

@xetorixik sorry, but I don't understand: are you experiencing my same problem or not? Cheers

mcspr commented 6 years ago

@andreagrandi Why do you specify url with .well-known in it? Shouldn't it be https://1.1.1.1/dns-query and https://1.0.0.1/dns-query? I don't see any mention of it here

ghost commented 6 years ago

hmm, good question @mcspr I was curious about the URL when I was following the instructions. Not sure if it would cause the fault but worth fixing.

andreagrandi commented 6 years ago

@mcspr I saw that url here https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/ which one should I use instead? thanks

mcspr commented 6 years ago

@andreagrandi Urls that I mentioned, or just do not use --upstream ... because they are builtin as default choices: https://github.com/cloudflare/cloudflared/blob/9135a4837c16a8f1f0dc4073d339db0a03d6579f/cmd/cloudflared/main.go#L264-L269

@AlexaBible You can test that curl 'https://1.1.1.1/.well-known/dns-query?ct=application/dns-json&name=cloudflare-dns.com&?type=A' does not work.

andreagrandi commented 6 years ago

I just removed the --upstream parameters and it seems to be running fine! I will keep an eye on this and let you know if I still get high CPU usage. Thanks for helping!

andreagrandi commented 6 years ago

False alarm :( previously I was misled because I had my VPN on and it bypass the system DNS. Disconnecting the VPN (and using pihole DNS) the cloudflared doesn't work at all for me. I can't see any error in the console but no address is being resolved. I reverted using a fix DNS and deactivated cloudflared.

cvocvo commented 6 years ago

I'm getting this issue with Cloudflared version 2018-7.2 on a Raspberry Pi 2 Model B. Did anyone resolve this somehow? Experiencing the same random failures + need to reboot to get it working again. (Undesirable if I'm not home and can't reboot the Pi for other users)

ghost commented 6 years ago

Personally, I reinstalled but followed the official guide. I then just set the dns in the GUI. From then on it worked for me.

cvocvo commented 6 years ago

@AlexaBible thanks for the quick response—can link me to the guide?

angussidney commented 6 years ago

@cvocvo I believe he's referring to this page.

I have also encountered the issue, I am yet to attempt Alexa's suggestion but I will post an update here if it works

it-can commented 6 years ago

Also getting this problem sometimes, seems to happen when I reboot my router and the internet connection is not available (running a pihole with cloudflared). Even when the router is running again after the reboot the cloudflared client keeps spitting out errors, a restart of cloudflared seems to fix this... Is there a more elegant way to fix this?

Oct 23 18:38:33 raspberrypi cloudflared[360]: time="2018-10-23T18:38:33+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 23 18:38:33 raspberrypi cloudflared[360]: time="2018-10-23T18:38:33+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

cloudflared --version cloudflared version 2018.10.3 (built 2018-10-10-2045 UTC)

soulawaker commented 6 years ago

In my case, it stopped when my laptop connected to a VPN network. I must restart it manually. Error messages below:

$ systemctl status cloudflared.service
...
...msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
...msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
notr1ch commented 6 years ago

FWIW, this seems specific to cloudflared rather than one of the Go packages. I've been using dnscrypt-proxy (also written in Go) as an alternative and it has been running flawlessly.

soulawaker commented 6 years ago

I tried dnscrypt-proxy. It was nice for a while when waking from sleep, recovering from screen lock, switching between LAN and VPN. But sometimes it has been same trouble, especially when switching between LAN and VPN for a long interval. I must restart it also. I'm puzzled why it happen.

callifo commented 6 years ago

Getting these errors on my pi too after an internet reboot or drop-out and reconnect,

failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Strange thing is, I used this on another RPI also running cloudflared too and it doesnt not have the same issue, it seems to noticed when the internet drops out and the HTTPS connection re-establishes fine. Configuration of the proxy is exactly the same.

After a manual service restart, things are up and running as normal.

andreagrandi commented 6 years ago

@callifo I would be curious to re-try again the whole setup with a more recent version of cloudflared. Last time I tried it was May 2018.

callifo commented 6 years ago

Yeah I'm running 2018.10.0 which I think is still the latest, and it still seems to behave the same way you described. A lot of issues all around the same date May-ish this year but not much since.

Was hoping for a fix but its been going on for a while. Might have to go low brow and use bash to detect when google.com stops resolving and reboot the daemon.

hydrajump commented 5 years ago

I've compiled version 2018.12.1 for mipsle and I was happy that it was working on my Ubiquiti ERX, but after multiple complaints that DNS resolution stopped working I had to remove cloudflared and just do plain DNS to 1.1.1.1 :(

I get the same "failed to connect to an HTTPS backend" and also netstat reports multiple sockets in CLOSE_WAIT state.

callifo commented 5 years ago

Retested 2018.12.1 and still having the same issue. Fortunately my internet stays up for months at a time so its not too inconvenient.

Its really odd, as I have two RPI's running it in different locations and both are configured the same with cloudflared, and only the local one has the issue. The other can handle internet drops without a problem.

neilzuk commented 5 years ago

I have also been having this issue with 2018.12.1 when the internet connection drops. I have to restart the cloudflared service for it to start resolving again on my RPi.

cloudflared[2809]: time="2019-01-21T18:55:01Z" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https:
acrogenesis commented 5 years ago

I'm also getting lots of this errors:

ERRO[0614] failed to connect to an HTTPS backend "https://1.1.1.1/dns-query"  error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: read tcp 192.168.86.113:50946->1.1.1.1:443: read: connection reset by peer"

I'm on mac installed via brew running VERSION: 2018.12.1 (built 2018-12-11-2047 UTC)

jankais3r commented 5 years ago

Having the same issue as everybody else:

pi@pihole:~ $ systemctl status cloudflared
● cloudflared.service - cloudflared DNS over HTTPS proxy
   Loaded: loaded (/lib/systemd/system/cloudflared.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2019-02-09 21:34:04 CET; 1min 4s ago
 Main PID: 1433 (cloudflared)
   CGroup: /system.slice/cloudflared.service
           └─1433 /usr/local/bin/cloudflared proxy-dns --port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query

Feb 09 21:31:38 pihole cloudflared[346]: time="2019-02-09T21:31:38+01:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

I am running quite old version as it seems to be the last one compatible with Raspberry 1st gen.

pi@ pihole:~ $ cloudflared -v
cloudflared version 2018.7.2 (built 2018-07-13-1701 UTC)

Update: I just built from the latest source to get around the segfault in the official binary, still getting the error.

luckylittle commented 5 years ago

+1

Feb 13 18:57:13 localhost cloudflared[5807]: time="2019-02-13T18:57:13+11:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
cloudflared version 2018.12.1 (built 2018-12-11-2030 UTC)
EPiC-APOC commented 5 years ago

Same thing here, it works but when I reboot my Raspberry (Pi-Hole) I also get the mentioned errors: level=error msg="failed to connect to an HTTPS backend \"[https://1.1.1.1/dns-query\](https://1.1.1.1/dns-query)"" error="failed to perform an HTTPS request etcetcetc

When I restart cloudflared everything is fine again, really annoying (bug?) :'(

I'm runing this on a Raspberry Pi 2B+ with the latest up2date Raspbian Stretch Lite and latest up2date Pihole with the current latest Cloudflare argo-tunnel:

wget https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-arm.tgz
tar -xvzf cloudflared-stable-linux-arm.tgz
cp ./cloudflared /usr/local/bin
chmod +x /usr/local/bin/cloudflared
cloudflared -v

(I also tried other argo-tunnel versions, but without any luck)

For me it does not make any difference if I use this tutorial https://docs.pi-hole.net/guides/dns-over-https/ or this one https://bendews.com/posts/implement-dns-over-https/ though.

...I also tried this next one (Cloudflare+Cloud9 DoH), just because we can ;-p https://www.reddit.com/r/pihole/comments/aoezvx/how_to_install_pihole_with_dns_over_https_cloud/ ...and also this other option/tutorial: https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/ ...You probably all ready might have guessed it, no luck either 👎

I can workaround it with this next line: sudo systemctl restart cloudflared 👍 If I check the status with: sudo systemctl status cloudflared everything is fine/okay, but I need to do this every time after I reboot something, it does look like a bug in my opinion. (or isn't it a bug, but an undocumented feature ;-p lol)

ohthehugemanatee commented 5 years ago

Reporting the same problem: when my internet connection drops, I get the expected messages in logs (running on Raspberry Pi 3):

Mar 12 09:29:16 raspberrypi cloudflared[488]: time="2019-03-12T09:29:16Z" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/.well-known/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 12 09:29:16 raspberrypi cloudflared[488]: time="2019-03-12T09:29:16Z" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/.well-known/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Mar 12 09:29:16 raspberrypi cloudflared[488]: time="2019-03-12T09:29:16Z" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/.well-known/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/.well-known/dns-query: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
...

My first guess is that cloudflared might try to re-use existing connections, with a timeout between re-establishing the connection. In that case it would make a difference how LONG the internet connection was down (in my case, more than 2 hours) - a short disruption might be handled by TCP error correction.

BUT after the connection is restored it still doesn't return DNS responses. I can fix it by pinging 1.0.0.1 (not restarting cloudflared) . This makes me suspect a lower layer issue - routing, perhaps?

p-doyle commented 5 years ago

I am having the exact same issue as ohthehugemanatee. Whenever I reboot my router cloudflared stops working. Once the router is back up and it regains internet access I have to restart the cloudflared service for it to start working again.

spantaleev commented 5 years ago

I've experienced this problem a few times as well and I suspect it's also triggered by a temporary network failure.

I've observed the exact same problem on another Golang project of mine. On that project, we've noticed that switching the device's connection (say between Wi-Fi and Ethernet) while the daemon is running, causes it to break. It's likely trying to re-use some old TCP connection or something.. All such attempts fail for quite a while (about 30 minutes) before it auto-recovers.

Our workaround was to disable HTTP Keep Alive for the http.Transport that gets passed to http.Client. Example:

httpClient := &http.Client{
    Transport: &http.Transport{
        DisableKeepAlives: true,
    },
}

Looking at the cloudflared code, it seems like tunneldns/https_upstream.go is responsible for creating and configuring the HTTP transport and that's where a similar workaround may be employed.

.. unless there's a better way to fix the problem, which won't completely eliminate Keep-Alive.

spaelling commented 5 years ago

Any update on this? I am not able to use the newest version on my raspberry pi (pihole) due to the segmentation fault error, but it would be nice to know that there is a working version. I had this working at one point, will test if this was due to restarting the cloudflared service as someone mentions.

Schmuuu commented 5 years ago

I had to disable the service on my system. I already created a restart script, which ran every 10min and restarted the service in case it lost the connection again and couldn't resolve any DNS requests. However, the outages were far too often:

Problem with service cloudflared-dns! Restarting service...
Apr 26 23:06:43 nextcloud cloudflared[21640]: time="2019-04-26T23:06:43+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Apr 26 23:06:43 nextcloud cloudflared[21640]: time="2019-04-26T23:06:43+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Restart successful

Problem with service cloudflared-dns! Restarting service...
Apr 27 01:35:23 nextcloud cloudflared[31489]: time="2019-04-27T01:35:23+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Apr 27 01:35:23 nextcloud cloudflared[31489]: time="2019-04-27T01:35:23+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Apr 27 01:35:23 nextcloud cloudflared[31489]: time="2019-04-27T01:35:23+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Restart successful

Problem with service cloudflared-dns! Restarting service...
Apr 27 19:11:21 nextcloud cloudflared[27077]: time="2019-04-27T19:11:21+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Apr 27 19:11:21 nextcloud cloudflared[27077]: time="2019-04-27T19:11:21+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Restart successful

Problem with service cloudflared-dns! Restarting service...
Apr 27 23:04:31 nextcloud cloudflared[31482]: time="2019-04-27T23:04:31+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Restart successful

Problem with service cloudflared-dns! Restarting service...
Apr 28 00:21:19 nextcloud cloudflared[23845]: time="2019-04-28T00:21:19+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Apr 28 00:21:19 nextcloud cloudflared[23845]: time="2019-04-28T00:21:19+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Apr 28 00:21:19 nextcloud cloudflared[23845]: time="2019-04-28T00:21:19+02:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="returned status code 400"
Restart successful

Too bad actually.

EPiC-APOC commented 5 years ago

Did someone find a solution to this already ??? It's still an issue, even with the latest argo-tunnel (cloudflare) and latest Pi-hole 4.3 (I'm running this on a Raspberry Pi 2B+ with an up2date Raspbian Stretch Lite).

I tried these, but all with same result: https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/ https://oliverhough.cloud/blog/configure-pihole-with-dns-over-https/ https://bendews.com/posts/implement-dns-over-https/ https://docs.pi-hole.net/guides/dns-over-https/ and also tried using yml as stated here (pt.5 and below): https://developers.cloudflare.com/1.1.1.1/dns-over-https/cloudflared-proxy/

it does work well for a while but than after 'periode-X' sudo systemctl status cloudflared does again return: level=error msg="failed to connect to an HTTPS backend etcetcetc :'( when I run sudo systemctl restart cloudflared things a working well again (no failed to etc.)

spaelling commented 5 years ago

I installed this on an Ubuntu server and pointed my Pihole at it. Worked without any issues for weeks now.

it-can commented 5 years ago

@spaelling try disconnecting the internet to the machine...

billthefarmer commented 5 years ago

I have posted a comment https://github.com/cloudflare/cloudflared/issues/38#issuecomment-498166512. When built with latest go 1.12.5 from latest sources (babcd9f) on a Pi 1B (BCM2835), runs out of file handles after a while and stops working.

hasmar04 commented 4 years ago

Thought I'd add that I still get the https backend error on 2019.11.3 running on a Raspberry Pi 3B+ with Pihole. Has anyone found a solution since the last comment?

emily-pesce commented 4 years ago

I still get it too. When my internet connection drops I have to reboot the Pi to get DNS lookups working again.

~ $ cloudflared -v cloudflared version 2019.11.0 (built 2019-11-07-1631 UTC)

hasmar04 commented 4 years ago

I find that just restarting the systemd service works for me. Sometimes I have to do it more than once though. This also allows my pihole and PiVPN to continue functioning.

insasho commented 4 years ago

Workaround: Consider using dnscrypt-proxy instead (https://github.com/DNSCrypt/dnscrypt-proxy). It can do the same DNS proxying that cloudflared does, and more.

andreagrandi commented 4 years ago

Workaround: Consider using dnscrypt-proxy instead (https://github.com/DNSCrypt/dnscrypt-proxy). It can do the same DNS proxying that cloudflared does, and more.

👍 that's exactly how I "solved" the problem for good. Been using it for weeks and it never crashed.

jPPuetz commented 4 years ago

I can still confirm that this is a big issue. I'm running a pihole with cloudflared on RPI 3B+ and if this bug appears the RPI stops responding to everything. Luckely I have now configured the watchdog and the watchdog will force a reboot if my RPI got stuck but this isn't a solution. I will check out dnscrypt-proxy too.

cloudflared version 2019.11.3 (built 2019-11-20-2250 UTC)

emily-pesce commented 4 years ago

@jPPuetz which watchdog? Can you share the configuration?

jPPuetz commented 4 years ago

@jPPuetz which watchdog? Can you share the configuration?

Just basic configuration of watchdog.conf (set device to /dev/watchdog and timeout to 14 secs). Information taken from here: https://archlinuxarm.org/wiki/Raspberry_Pi (search for watchdog) https://aur.archlinux.org/packages/watchdog/ and important here: https://raspberrypi.stackexchange.com/questions/92462/installing-watchdog-daemon-on-arch

yssource commented 4 years ago
Jan 12 19:51:22 jimmy-arch systemd[1]: Started Argo Tunnel.
Jan 12 19:52:53 jimmy-arch cloudflared[578]: time="2020-01-12T19:52:53+08:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: >
Jan 12 19:52:58 jimmy-arch cloudflared[578]: time="2020-01-12T19:52:58+08:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: >
Jan 12 19:52:58 jimmy-arch cloudflared[578]: time="2020-01-12T19:52:58+08:00" level=error msg="failed to connect to an HTTPS backend \"https://1.1.1.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.1.1.1/dns-query: >
Jan 12 19:53:03 jimmy-arch cloudflared[578]: time="2020-01-12T19:53:03+08:00" level=error msg="failed to connect to an HTTPS backend \"https://1.0.0.1/dns-query\"" error="failed to perform an HTTPS request: Post https://1.0.0.1/dns-query: >
froedel commented 4 years ago

Same thing here, it works but when I reboot my Raspberry (Pi-Hole) I also get the mentioned errors: level=error msg="failed to connect to an HTTPS backend \"[https://1.1.1.1/dns-query\](https://1.1.1.1/dns-query)"" error="failed to perform an HTTPS request etcetcetc

When I restart cloudflared everything is fine again, really annoying (bug?) :'(

I can workaround it with this next line: sudo systemctl restart cloudflared +1 If I check the status with: sudo systemctl status cloudflared everything is fine/okay, but I need to do this every time after I reboot something, it does look like a bug in my opinion. (or isn't it a bug, but an undocumented feature ;-p lol)

+1

Raspi 3B+

cloudflared version 2019.9.0 (built 2019-09-06-0334 UTC)

in combination with latest pihole

Pi-hole version is v4.3.2 (Latest: v4.3.2) AdminLTE version is v4.3 (Latest: v4.3.2) FTL version is v4.3.1 (Latest: v4.3.1)

davideaicardi commented 4 years ago

Does anybody know if there is a permanent fix? I temporary solved this restarting the service, but I can't relaying on something that may broke all connections in the house

jankais3r commented 4 years ago

My permanent fix was to switch to dnscrypt-proxy 2 for my DoH needs. Not a single connection drop in the past 1.5 years.