Intermittent network access errors in 4.24.2 fixed by downgrading to 4.24.0

abulka commented 1 year ago

Description

Running a web app deployed via dokku running in docker:

getting 404s intermittently,
getting intermittent Whitelabel Error Page errors
```
This application has no explicit mapping for /error, so you are seeing this as a fallback.
Sat Oct 14 12:29:06 UTC 2023
There was an unexpected error (type=Not Found, status=404).
```
Refreshing any web page with the errror a few times causes it to work. nginx access logs show nothing when 404 errors occur, suggesting its a docker network issue?

The app is a django app, deployed into a container by dokku. Dokku uses nginx.

App was working fine in 4.24.0 - problems begain when upgraded to docker desktop for mac 4.24.2

Downgrading from 4.24.2 -> 4.24.0 fixed the problem. (By uninstalling docker desktop and re-installing 4.24.0, then rebuilding the containers).

Reproduce

Install dokku in docker

docker container run \
  --env DOKKU_HOSTNAME=dokku.me \
  --env DOKKU_HOST_ROOT=/var/lib/dokku/home/dokku \
  --env DOKKU_LIB_HOST_ROOT=/var/lib/dokku/var/lib/dokku \
  --name dokku \
  --publish 3022:22 \
  --publish 80:80 \
  --publish 8443:443 \
  --volume /var/lib/dokku:/mnt/dokku \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume ~/.ssh:/tmp/ssh \
  --restart on-failure:3 \
  dokku/dokku:0.31.4

Install ssh keys into dokku.

Install my app

docker exec -it dokku bash
dokku apps:create gituml
dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres
dokku postgres:create gituml-database
dokku postgres:link gituml-database gituml
dokku network:create internal-network
dokku network:set gituml attach-post-deploy internal-network
dokku storage:ensure-directory gituml
dokku storage:mount gituml /var/lib/dokku/data/storage/gituml:/code/media

git remote add dokkume dokku@dokku.docker:gituml
git push dokkume slim_staging:master

Assuming /etc/hosts contains

127.0.0.1       gituml.dokku.me

Access http://gituml.dokku.me/ then browse through the pages of the app. They intermittently fail.

Expected behavior

web pages should respond at all times without errors

docker version

docker version (after repairing the situation and downgrading)
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:52 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Desktop 4.24.0 (122432)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

docker info (after repairing the situation and downgrading)
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 4
  Paused: 0
  Stopped: 2
 Images: 40
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 9
 Total Memory: 7.667GiB
 Name: docker-desktop
 ID: 439171b3-ddd9-4863-a872-579777d06785
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

FB17F6AE-AEB5-4004-9265-894AC3CB1F21/20231015054159

Additional Info

mac ventura on Macbook Air M2
rebuilding image and container didn't help
restarting docker didn't help
restarting computer didn't help
running same app via docker compose (without nginx and without dokku) did not exhibit the problem

bsousaa commented 1 year ago

can check if this workaround fixes the issue? https://github.com/docker/for-mac/issues/7008#issuecomment-1746653802

abulka commented 11 months ago

@bsousaa Sorry I can't seem to reproduce the problem anymore, after re-installing docker and rebuilding images and containers etc.

docker / for-mac