Connection errors with DropBox and Buienradar - Failed to establish a new connection: [Errno -3] Try again')

Issue

Hi there,

This is a continuation from troubleshooting DSMR missing temperatures . After enabling DEBUG logging, I'm seeing both the Buienradar and Dropbox connections failing. External connectivity into the container is perfectly fine; I can access the webpage both from the internal network and the internet.

The strange part is, sometimes restarting the Raspberry Pi host and/or the containers would fix the issue. I'd get temperatures in the graphs again and have dropbox backups, until I updated the images again and reprovisioned the containers.

Any suggestions where to start troubleshooting are highly appreciated!

Which version has the issue?

latest

What was the last working version?

No response

What type of installation are you running (architecture)?

No response

Anything in the logs that might be useful?

2021-12-16 14:06:46,527 ERROR    schedule     execute_scheduled_processes      37 | (ConnectionError) dsmr_dropbox.services.run errored: HTTPSConnectionPool(host='api.dropboxapi.com', port=443): Max retries exceeded with url: /2/users/get_space_usage (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb21e6d10>: Failed to establish a new connection: [Errno -3] Try again'))
2021-12-16 14:07:22,525 ERROR    schedule     execute_scheduled_processes      37 | (ConnectionError) dsmr_dropbox.services.run errored: HTTPSConnectionPool(host='api.dropboxapi.com', port=443): Max retries exceeded with url: /2/users/get_space_usage (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb21e5ba0>: Failed to establish a new connection: [Errno -3] Try again'))
2021-12-16 14:07:43,947 ERROR    services     run                              22 | Buienradar: Failed to read API: HTTPSConnectionPool(host='data.buienradar.nl', port=443): Max retries exceeded with url: /2.0/feed/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb25011e0>: Failed to establish a new connection: [Errno -3] Try again'))

Additional information

Docker-compose file: docker-compose.txt

Hi, it seems to me that internet is unavaile when in such a state. You can verify this by executing the following for example:

docker exec dsmr bash -c "curl -sI https://www.buienradar.nl"

It should return the response code and some text... if not, then something is wrong with the internet connection within your container.

Or:

docker exec dsmr bash -c "curl -s https://www.buienradar.nl"

For the complete output of the page in text format.

Indeed, no output. Could this be a DNS thing? The resolve.conf output doesn't make sense to me, but I must admit that is my first troubleshooting session in docker so not quite sure what to expect here.

Some extra output:

pi@pi4b01:~ $ docker exec dsmr bash -c "ping 8.8.8.8"
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=17.541 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=15.613 ms
^C

pi@pi4b01:~ $ docker exec dsmr bash -c "ifconfig"
eth0      Link encap:Ethernet  HWaddr 02:42:AC:14:00:03
          inet addr:172.20.0.3  Bcast:172.20.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38168 errors:0 dropped:0 overruns:0 frame:0
          TX packets:63194 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:17516923 (16.7 MiB)  TX bytes:23229652 (22.1 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:5556 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5556 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:498385 (486.7 KiB)  TX bytes:498385 (486.7 KiB)
pi@pi4b01:~ $ docker exec dsmr bash -c "route"
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.20.0.1      0.0.0.0         UG    0      0        0 eth0
172.20.0.0      *               255.255.0.0     U     0      0        0 eth0
pi@pi4b01:~ $ docker network inspect dsmr-reader_default
[
    {
        "Name": "dsmr-reader_default",
        "Id": "c1cffaf61f52325158cde2577a2ae77fdd36abb1fdfa51c6f07edeaa026eb337",
        "Created": "2021-04-20T10:22:07.13267126+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "9319728a214d2de2547962e783d0b1c3653e54c387989499e1d4be421cbf6762": {
                "Name": "dsmrdb",
                "EndpointID": "b234b035eae3dda06f7031282c3d7c10b6a4b33b19d8802c0f0d2cd3b4955869",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            },
            "f1acd989f15fc90d74a6fbaddb6f26f25072d7c9f1fea8a976dcc2519ce37512": {
                "Name": "dsmr",
                "EndpointID": "a4a1d1ef84f8d67006839011094d3138fce7914f8a1e1fa11ef931955cfe7586",
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "dsmr-reader"
        }
    }
]
pi@pi4b01:~ $ docker exec dsmr bash -c "cat /etc/resolv.conf"
search net
nameserver 127.0.0.11
options ndots:0

Most probably you're using the default bridge network?

On what kind of host are you running Docker? A nas, which type, Docker verison? AdBlockers (internal DNS forwarder), VPN or proxy stuff running perhaps?

If you can provide me that info, I might be able to help you out.

It has something todo with the daemon and it's not an issue within the container itself.

Much appreciated! I'm running it on a Raspberry Pi 4B:

pi@pi4b01:~ $ uname -a
Linux pi4b01 5.10.63-v8+ #1496 SMP PREEMPT Wed Dec 1 15:59:46 GMT 2021 aarch64 GNU/Linux
pi@pi4b01:~ $ docker info
Containers: 5
 Running: 5
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 18.09.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 1.0.0~rc6+dfsg1-3
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
Kernel Version: 5.10.63-v8+
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 7.63GiB
Name: pi4b01
ID: NZ73:6B4M:EAH7:WN3H:QUFY:RIQ2:6IL5:JHM5:5X3E:PR25:KUDK:ML2C
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support

pi@pi4b01:~ $ docker network list
NETWORK ID          NAME                  DRIVER              SCOPE
9c00acfe436f        bridge                bridge              local
c1cffaf61f52        dsmr-reader_default   bridge              local
d5bb9b929ffb        host                  host                local
f5b196f1c497        none                  null                local
88b294b4dcca        unifi_default         bridge              local

pi@pi4b01:~ $ docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                  PORTS                                                                                                                                                                                                                                            NAMES
f1acd989f15f        xirixiz/dsmr-reader-docker:latest      "/bin/bash -c /app/r…"   21 hours ago        Up 21 hours (healthy)   0.0.0.0:7777->80/tcp, 0.0.0.0:7779->443/tcp                                                                                                                                                                                                      dsmr
9319728a214d        postgres:13-alpine                     "docker-entrypoint.s…"   21 hours ago        Up 21 hours (healthy)   5432/tcp                                                                                                                                                                                                                                         dsmrdb
<snip>
10233939cb74        pihole/pihole:latest                   "/s6-init"               4 days ago          Up 3 days (healthy)                                                                                                                                                                                                                                                      pihole

There is a Pihole in the network (also running as a container), which forwards out to Cloudflare.

If I do a lookup on the host itself, I get a reply:

pi@pi4b01:~ $ nslookup www.google.com
Server:         192.168.120.210
Address:        192.168.120.210#53

Non-authoritative answer:
Name:   www.google.com
Address: 172.217.168.196
Name:   www.google.com
Address: 2a00:1450:400e:811::2004

If I do a DNS resolve in the dsmr container, I don't:

pi@pi4b01:~ $ docker exec dsmr bash -c "nslookup www.google.com"
;; connection timed out; no servers could be reached

DSMR containers are attached to a dsmr-only network:

pi@pi4b01:~ $ docker network inspect dsmr-reader_default
[
    {
        "Name": "dsmr-reader_default",
        "Id": "c1cffaf61f52325158cde2577a2ae77fdd36abb1fdfa51c6f07edeaa026eb337",
        "Created": "2021-04-20T10:22:07.13267126+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "9319728a214d2de2547962e783d0b1c3653e54c387989499e1d4be421cbf6762": {
                "Name": "dsmrdb",
                "EndpointID": "9736e27c20e6c6adcf338b41019b13c26c042c1afdb98bb709a0aba4450eac1e",
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            },
            "f1acd989f15fc90d74a6fbaddb6f26f25072d7c9f1fea8a976dcc2519ce37512": {
                "Name": "dsmr",
                "EndpointID": "50933ac5508c0ab17a068d0d910e18b35203d5434c5e9bc3e0be9db23f926fb6",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "dsmr-reader"
        }
    }
]

And this is the DNS server in the container itself, which looks alright as far as I can google :) :

pi@pi4b01:~/docker/dsmr-reader $ docker exec dsmr bash -c "cat /etc/resolv.conf"
search net
nameserver 127.0.0.11
options ndots:0

I've changed the static DNS on the PI host to the network router in the dhcpcd.conf, restarted dhcpcd, restarted the dsmr container, and voila:

pi@pi4b01:~/docker/dsmr-reader $ docker exec dsmr bash -c "nslookup www.google.com"
Server:         127.0.0.11
Address:        127.0.0.11:53

Non-authoritative answer:
Name:   www.google.com
Address: 142.250.179.132

Non-authoritative answer:
Name:   www.google.com
Address: 2a00:1450:400e:801::2004

Then change it back again, and it's broken again:

pi@pi4b01:~/docker/dsmr-reader $ nano /etc/dhcpcd.conf
pi@pi4b01:~/docker/dsmr-reader $ sudo service dhcpcd restart
pi@pi4b01:~/docker/dsmr-reader $ cat /etc/resolv.conf
# Generated by resolvconf
search net
nameserver 192.168.120.210
nameserver fd00::1:22d5:bfff:feb4:f2c2
nameserver fe80::22d5:bfff:feb4:f2c2%eth0
nameserver 2001:b88:1002::10
nameserver 2001:b88:1202::10
nameserver 2001:730:3e42:1000::53
pi@pi4b01:~/docker/dsmr-reader $ docker restart dsmr
dsmr
pi@pi4b01:~/docker/dsmr-reader $ docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                    PORTS                                                                                                                                                                                                                      NAMES
f1acd989f15f        xirixiz/dsmr-reader-docker:latest      "/bin/bash -c /app/r…"   22 hours ago        Up 31 seconds (healthy)   0.0.0.0:7777->80/tcp, 0.0.0.0:7779->443/tcp                                                                                                                                                                                dsmr
9319728a214d        postgres:13-alpine                     "docker-entrypoint.s…"   22 hours ago        Up 17 minutes (healthy)   5432/tcp                                                                                                                                                                                                                   dsmrdb
10233939cb74        pihole/pihole:latest                   "/s6-init"               4 days ago          Up 17 minutes (healthy)                                                                                                                                                                                                                              pihole
pi@pi4b01:~/docker/dsmr-reader $ docker exec dsmr bash -c "nslookup www.google.com"
;; connection timed out; no servers could be reached

So there's a workaround, but I'm not quite sure yet why this is/was an intermittent problem, sometimes just solved with a reboot 😕 Any suggestions?

Alright, so to confirm; I've redeployed the pihole container (w/ same settings), left the DSMR containers alone... and it works again with DNS resolving pointed to the Pihole. So this is definitely not a DSMR container issue, rather something else in my docker/pihole config.

I'll see if the issue reoccurs. In any case, thank you for your help and some pointers in troubleshooting commands! 👍

Great! Sorry I can't help out much more, it's all so dependent of your infra behind Docker. I also had some issues with Docker and bridge networking. You could also consider to look into macvlan, use host networking, or maybe something else. Could also some firewall rule in your network is blocking DNS traffic f.e. from Docker networks. Some users experienced that in a Synology setup. Anyway, feel free to ask any question.

xirixiz / dsmr-reader-docker