search

nickjj / docker-rails-example

A production ready example Rails app that's using Docker and Docker Compose.
MIT License
941 stars 185 forks source link

Unable to establish DB connection from web server #68

Closed JeremiahChurch closed 9 months ago

JeremiahChurch commented 9 months ago

I'm trying to fire up the base example from the readme. All the containers build and seemingly start correctly. However the web server returns ConnectionNotEstablished when trying to connect to postgres and it looks like the workers can't connect to redis (I haven't looked in to that as postgres really blocks anything.)

I can ping and telnet to the postgres server pg port from the host machine. It looks like ping isn't installed on any of the containers so I can't test directly from the webserver.

Any ideas or pointers would be GREATLY appreciated. I'd love to move a bunch of my rails apps over to docker!

Incoming pile of version & environment data - please let me know if anything else would be helpful

app is top of main from yesterday host is ubuntu 20 I have a few other dockers running with networking but nothing seems to collide with the hellorails_default

jchurch@ubuntu:~/hellorails$ docker network ls
NETWORK ID     NAME                      DRIVER    SCOPE
1565655ad8a9   bridge                    bridge    local
36869fcbef8f   hellorails_default        bridge    local
d16411050e7a   host                      host      local
1052fedf8e1e   myapp_default             bridge    local
dabb136ba7eb   none                      null      local
689c8988682a   psychic-pancake_default   bridge    local
jchurch@ubuntu:~/hellorails$ docker ps
CONTAINER ID   IMAGE                                   COMMAND                  CREATED          STATUS                    PORTS                                                     NAMES
b4bd28991a21   hellorails-js                           "yarn build"             46 minutes ago   Up 46 minutes                                                                       hellorails-js-1
e607f3b1f559   hellorails-cable                        "puma -p 28080 cable…"   46 minutes ago   Up 46 minutes             8000/tcp, 0.0.0.0:28080->28080/tcp, :::28080->28080/tcp   hellorails-cable-1
ddb52e2655f6   hellorails-web                          "/app/bin/docker-ent…"   46 minutes ago   Up 46 minutes (healthy)   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp                 hellorails-web-1
f9bb4938ab11   hellorails-css                          "yarn build:css"         46 minutes ago   Up 46 minutes                                                                       hellorails-css-1
28a2282b0a54   postgres:15.4-bookworm                  "docker-entrypoint.s…"   15 hours ago     Up 46 minutes             5432/tcp                                                  hellorails-postgres-1
38517dca762e   redis:7.0.12-bookworm                   "docker-entrypoint.s…"   15 hours ago     Up 46 minutes             6379/tcp                                                  hellorails-redis-1
092da6e0dbea   ghcr.io/beeper/linkedin:latest          "/opt/linkedin-matri…"   7 days ago       Up 46 minutes             0.0.0.0:29329->29329/tcp, :::29329->29329/tcp             modest_moser
46ac294cb409   dock.mau.dev/mautrix/whatsapp:latest    "/docker-run.sh"         7 weeks ago      Up 46 minutes             0.0.0.0:29318->29318/tcp, :::29318->29318/tcp             fervent_yalow

jchurch@ubuntu:~/hellorails$ docker compose version
Docker Compose version v2.20.2-desktop.1
jchurch@ubuntu:~/hellorails$ docker network inspect hellorails_default
[
    {
        "Name": "hellorails_default",
        "Id": "36869fcbef8fb291c6a99063f612ddae8ae48f2f2e756de0ab7360e189a89d09",
        "Created": "2023-09-12T20:30:44.800489461-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "28a2282b0a544a213285da101e09f0888f852fb52ede1852007f232b7556b70b": {
                "Name": "hellorails-postgres-1",
                "EndpointID": "fe05c4ad329531fea86ddda820e79b9d4eb6498631deb34b95a8905864b74684",
                "MacAddress": "02:42:ac:14:00:04",
                "IPv4Address": "172.20.0.4/16",
                "IPv6Address": ""
            },
            "38517dca762ea5ffdfea7c7b69cae7306a25bb88b619d4ba1d9e0a58ebb97667": {
                "Name": "hellorails-redis-1",
                "EndpointID": "ec6ac6039498873453ce45ee2e7ee012949ccdd7a2707c1ac3fb619502a163e3",
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            },
            "b4bd28991a21652b934ad85e67f32d314b95133a57d30e132b825743504c65ad": {
                "Name": "hellorails-js-1",
                "EndpointID": "d30e5252656e8d5ae9a3b00082f1c12b451c0da801c0d24edb4156902390c34a",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            },
            "ddb52e2655f6c402fb921460cca3b893d6d8e77b131a2cb8abdd32ca507588b8": {
                "Name": "hellorails-web-1",
                "EndpointID": "939bab609d4ee25d3874750fab8191cf78e5dbdee7ffaa49611c78ee8d7aebb2",
                "MacAddress": "02:42:ac:14:00:06",
                "IPv4Address": "172.20.0.6/16",
                "IPv6Address": ""
            },
            "e607f3b1f55997d58fb4942f99415901172f26149c9cbcf8e6a1e5539ba72a06": {
                "Name": "hellorails-cable-1",
                "EndpointID": "879ca6c02f4a204cdc58ec9eb2a0fd9626e5ea72382ee0932d562c93029f1282",
                "MacAddress": "02:42:ac:14:00:07",
                "IPv4Address": "172.20.0.7/16",
                "IPv6Address": ""
            },
            "f9bb4938ab11986028861fc22de542910936e57132bc3ef59a6d0e2b5290f729": {
                "Name": "hellorails-css-1",
                "EndpointID": "dde1d71cfda6bb40361b50cdcc690bf892b5464b566b7f2c1d68c6452ee76775",
                "MacAddress": "02:42:ac:14:00:05",
                "IPv4Address": "172.20.0.5/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "hellorails",
            "com.docker.compose.version": "2.20.2"
        }
    }
]
jchurch@ubuntu:~/hellorails$ telnet 172.20.0.4 5432
Trying 172.20.0.4...
Connected to 172.20.0.4.
Escape character is '^]'.
^CConnection closed by foreign host.

app build/start output

jchurch@ubuntu:~$ cd hellorails/
direnv: export +COMPOSE_PROFILES +COMPOSE_PROJECT_NAME +DOCKER_BUILDKIT +DOCKER_CABLE_PORT_FORWARD +DOCKER_RESTART_POLICY +DOCKER_WEB_HEALTHCHECK_TEST +DOCKER_WEB_PORT_FORWARD +DOCKER_WEB_VOLUME +NODE_ENV +POSTGRES_PASSWORD +POSTGRES_USER +RAILS_ENV +RAILS_MAX_THREADS +SECRET_KEY_BASE +WEB_CONCURRENCY
jchurch@ubuntu:~/hellorails$ docker compose up --build
[+] Building 1.3s (38/67)                                                                                                                                                      docker:default
 => [worker internal] load build definition from Dockerfile                                                                                                                              0.0s
 => => transferring dockerfile: 32B                                                                                                                                                      0.0s
 => [web internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 32B                                                                                                                                                      0.0s
 => [js internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 32B                                                                                                                                                      0.0s
 => [css internal] load build definition from Dockerfile                                                                                                                                 0.0s
 => => transferring dockerfile: 32B                                                                                                                                                      0.0s
 => [worker internal] load .dockerignore                                                                                                                                                 0.0s
 => => transferring context: 35B                                                                                                                                                         0.0s
 => [cable internal] load build definition from Dockerfile                                                                                                                               0.0s
 => => transferring dockerfile: 32B                                                                                                                                                      0.0s
 => [web internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 35B                                                                                                                                                         0.0s
 => [js internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 35B                                                                                                                                                         0.0s
 => [css internal] load .dockerignore                                                                                                                                                    0.0s
 => => transferring context: 35B                                                                                                                                                         0.0s
 => [web internal] load metadata for docker.io/library/ruby:3.2.2-slim-bookworm                                                                                                          1.0s
 => [cable internal] load .dockerignore                                                                                                                                                  0.0s
 => => transferring context: 35B                                                                                                                                                         0.0s
 => [worker auth] library/ruby:pull token for registry-1.docker.io                                                                                                                       0.0s
 => [cable assets 1/9] FROM docker.io/library/ruby:3.2.2-slim-bookworm@sha256:b86f08332ea5f9b73c427018f28af83628c139567cc72823270cac6ab056c4dc                                           0.0s
 => [web internal] load build context                                                                                                                                                    0.1s
 => => transferring context: 65.29kB                                                                                                                                                     0.0s
 => [css internal] load build context                                                                                                                                                    0.0s
 => => transferring context: 65.29kB                                                                                                                                                     0.0s
 => [js internal] load build context                                                                                                                                                     0.0s
 => => transferring context: 65.29kB                                                                                                                                                     0.0s
 => [cable internal] load build context                                                                                                                                                  0.1s
 => => transferring context: 65.29kB                                                                                                                                                     0.0s
 => [worker internal] load build context                                                                                                                                                 0.1s
 => => transferring context: 65.29kB                                                                                                                                                     0.0s
 => CACHED [web assets 2/9] WORKDIR /app                                                                                                                                                 0.0s
 => CACHED [web assets 3/9] RUN bash -c "set -o pipefail && apt-get update   && apt-get install -y --no-install-recommends build-essential curl git libpq-dev   && curl -fsSL https://d  0.0s
 => CACHED [js assets 4/9] COPY --chown=ruby:ruby Gemfile* ./                                                                                                                            0.0s
 => CACHED [js assets 5/9] RUN bundle install                                                                                                                                            0.0s
 => CACHED [js assets 6/9] COPY --chown=ruby:ruby package.json *yarn* ./                                                                                                                 0.0s
 => CACHED [js assets 7/9] RUN yarn install                                                                                                                                              0.0s
 => CACHED [js assets 8/9] COPY --chown=ruby:ruby . .                                                                                                                                    0.0s
 => CACHED [js assets 9/9] RUN if [ "development" != "development" ]; then   SECRET_KEY_BASE=dummyvalue rails assets:precompile; fi                                                      0.0s
 => [css] exporting to image                                                                                                                                                             0.1s
 => => exporting layers                                                                                                                                                                  0.0s
 => => writing image sha256:1c048854f8388e92eaf65eef14095e6d0c752bb8620b230a52e00d4d7f63bbad                                                                                             0.0s
 => => naming to docker.io/library/hellorails-js                                                                                                                                         0.0s
 => => writing image sha256:c9569cd0330a46c310034f2da1bc24343e685a4b442ef0fc14da8a68534b5284                                                                                             0.0s
 => => naming to docker.io/library/hellorails-web                                                                                                                                        0.0s
 => => writing image sha256:50c83f892ac842aab7f8f8eaa5d4e84b4153c0b7e301fc4d5eb70d0dc69a6c4d                                                                                             0.0s
 => => naming to docker.io/library/hellorails-worker                                                                                                                                     0.0s
 => => writing image sha256:2207df3fa1bce63f81b545fb54b9ea3526fe24ca7ccbeab4b6c24af879e5909f                                                                                             0.0s
 => => naming to docker.io/library/hellorails-cable                                                                                                                                      0.0s
 => => writing image sha256:1e6359cab03064720efa15e9d91711f0f7c81a5b9fb3ebcc67967256bd400a12                                                                                             0.0s
 => => naming to docker.io/library/hellorails-css                                                                                                                                        0.0s
 => CACHED [web app 3/8] RUN apt-get update   && apt-get install -y --no-install-recommends build-essential curl libpq-dev   && rm -rf /var/lib/apt/lists/* /usr/share/doc /usr/share/m  0.0s
 => CACHED [web app 4/8] COPY --chown=ruby:ruby bin/ ./bin                                                                                                                               0.0s
 => CACHED [web app 5/8] RUN chmod 0755 bin/*                                                                                                                                            0.0s
 => CACHED [web assets 4/9] COPY --chown=ruby:ruby Gemfile* ./                                                                                                                           0.0s
 => CACHED [web assets 5/9] RUN bundle install                                                                                                                                           0.0s
 => CACHED [web assets 6/9] COPY --chown=ruby:ruby package.json *yarn* ./                                                                                                                0.0s
 => CACHED [web assets 7/9] RUN yarn install                                                                                                                                             0.0s
 => CACHED [web assets 8/9] COPY --chown=ruby:ruby . .                                                                                                                                   0.0s
 => CACHED [web assets 9/9] RUN if [ "development" != "development" ]; then   SECRET_KEY_BASE=dummyvalue rails assets:precompile; fi                                                     0.0s
 => CACHED [web app 6/8] COPY --chown=ruby:ruby --from=assets /usr/local/bundle /usr/local/bundle                                                                                        0.0s
 => CACHED [web app 7/8] COPY --chown=ruby:ruby --from=assets /app/public /public                                                                                                        0.0s
 => CACHED [web app 8/8] COPY --chown=ruby:ruby . .                                                                                                                                      0.0s
[+] Running 7/7
 ✔ Container hellorails-redis-1     Created                                                                                                                                              0.0s 
 ✔ Container hellorails-css-1       Recreated                                                                                                                                            0.1s 
 ✔ Container hellorails-js-1        Recreated                                                                                                                                            0.1s 
 ✔ Container hellorails-postgres-1  Created                                                                                                                                              0.0s 
 ✔ Container hellorails-cable-1     Recreated                                                                                                                                            0.1s 
 ✔ Container hellorails-web-1       Recreated                                                                                                                                            0.1s 
 ✔ Container hellorails-worker-1    Recreated                                                                                                                                            0.1s 
Attaching to hellorails-cable-1, hellorails-css-1, hellorails-js-1, hellorails-postgres-1, hellorails-redis-1, hellorails-web-1, hellorails-worker-1
hellorails-redis-1     | 1:C 13 Sep 2023 15:02:32.310 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
hellorails-redis-1     | 1:C 13 Sep 2023 15:02:32.310 # Redis version=7.0.12, bits=64, commit=00000000, modified=0, pid=1, just started
hellorails-redis-1     | 1:C 13 Sep 2023 15:02:32.310 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.310 * Increased maximum number of open files to 10032 (it was originally set to 1024).
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.310 * monotonic clock: POSIX clock_gettime
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.311 * Running mode=standalone, port=6379.
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.311 # Server initialized
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.311 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.312 * Loading RDB produced by version 7.0.12
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.312 * RDB age 45 seconds
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.312 * RDB memory usage when created 0.82 Mb
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.312 * Done loading RDB, keys loaded: 0, keys expired: 0.
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.312 * DB loaded from disk: 0.000 seconds
hellorails-redis-1     | 1:M 13 Sep 2023 15:02:32.312 * Ready to accept connections
hellorails-postgres-1  | 
hellorails-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
hellorails-postgres-1  | 
hellorails-postgres-1  | 
hellorails-postgres-1  | 2023-09-13 15:02:32.561 UTC [1] LOG:  starting PostgreSQL 15.4 (Debian 15.4-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
hellorails-postgres-1  | 2023-09-13 15:02:32.990 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
hellorails-postgres-1  | 2023-09-13 15:02:32.990 UTC [1] LOG:  listening on IPv6 address "::", port 5432

yarn run v1.22.19      | 
hellorails-postgres-1  | 2023-09-13 15:02:32.993 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
hellorails-postgres-1  | 2023-09-13 15:02:32.997 UTC [28] LOG:  database system was shut down at 2023-09-13 15:01:47 UTC
yarn run v1.22.19
hellorails-postgres-1  | 2023-09-13 15:02:33.030 UTC [1] LOG:  database system is ready to accept connections

hellorails-js-1        | 
hellorails-js-1        | $ ./run yarn:build
hellorails-js-1        | 
$ ./run yarn:build:css
hellorails-js-1        | [watch] build finished, watching for changes...
hellorails-js-1        | 
hellorails-css-1       | 
hellorails-css-1       | Rebuilding...
hellorails-cable-1     | [1] Puma starting in cluster mode...
hellorails-cable-1     | [1] * Puma version: 6.3.1 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")
hellorails-cable-1     | [1] *  Min threads: 1
hellorails-cable-1     | [1] *  Max threads: 1
hellorails-cable-1     | [1] *  Environment: development
hellorails-cable-1     | [1] *   Master PID: 1
hellorails-cable-1     | [1] *      Workers: 1
hellorails-cable-1     | [1] *     Restarts: (✔) hot (✖) phased
hellorails-cable-1     | [1] * Preloading application
hellorails-css-1       | 
hellorails-css-1       | Done in 407ms.
hellorails-web-1       | => Booting Puma
hellorails-web-1       | => Rails 7.0.7.2 application starting in development 
hellorails-web-1       | => Run `bin/rails server --help` for more startup options
hellorails-web-1       | [1] Puma starting in cluster mode...
hellorails-web-1       | 
hellorails-web-1       | [1] * Puma version: 6.3.1 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")
hellorails-web-1       | [1] *  Min threads: 1
hellorails-web-1       | [1] *  Max threads: 1
hellorails-web-1       | [1] *  Environment: development
hellorails-web-1       | [1] *   Master PID: 1
hellorails-web-1       | [1] *      Workers: 1
hellorails-web-1       | [1] *     Restarts: (✔) hot (✖) phased
hellorails-web-1       | [1] * Preloading application
hellorails-web-1       | [1] * Listening on http://0.0.0.0:8000
hellorails-web-1       | [1] Use Ctrl-C to stop
hellorails-web-1       | [1] ! WARNING: Detected running cluster mode with 1 worker.
hellorails-web-1       | [1] ! Running Puma in cluster mode with a single worker is often a misconfiguration.
hellorails-web-1       | [1] ! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead.
hellorails-web-1       | [1] ! Set the `silence_single_worker_warning` option to silence this warning message.
hellorails-web-1       | [1] - Worker 0 (PID: 10) booted in 0.0s, phase: 0
hellorails-worker-1    | 
hellorails-worker-1    | 
hellorails-worker-1    |                m,
hellorails-worker-1    |                `$b
hellorails-worker-1    |           .ss,  $$:         .,d$
hellorails-worker-1    |           `$$P,d$P'    .,md$P"'
hellorails-worker-1    |            ,$$$$$b/md$$$P^'
hellorails-worker-1    |          .d$$$$$$/$$$P'
hellorails-worker-1    |          $$^' `"/$$$'       ____  _     _      _    _
hellorails-worker-1    |          $:    ',$$:       / ___|(_) __| | ___| | _(_) __ _
hellorails-worker-1    |          `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
hellorails-worker-1    |                 $$:         ___) | | (_| |  __/   <| | (_| |
hellorails-worker-1    |                 $$         |____/|_|\__,_|\___|_|\_\_|\__, |
hellorails-worker-1    |               .d$$                                       |_|
hellorails-worker-1    |       
hellorails-worker-1    | 
hellorails-worker-1    | 2023-09-13T15:02:34.930Z pid=1 tid=2pt INFO: Booted Rails 7.0.7.2 application in development environment
hellorails-worker-1    | 2023-09-13T15:02:34.931Z pid=1 tid=2pt INFO: Running in ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
hellorails-worker-1    | 2023-09-13T15:02:34.931Z pid=1 tid=2pt INFO: See LICENSE and the LGPL-3.0 for licensing details.
hellorails-worker-1    | 2023-09-13T15:02:34.931Z pid=1 tid=2pt INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
hellorails-worker-1    | 2023-09-13T15:02:34.931Z pid=1 tid=2pt INFO: Sidekiq 7.1.2 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://redis:6379/1"}
hellorails-cable-1     | [1] * Listening on http://0.0.0.0:28080
hellorails-cable-1     | [1] ! WARNING: Detected 2 Thread(s) started in app boot:
hellorails-cable-1     | [1] ! #<Thread:0x00007f0df71ef558@DEBUGGER__::SESSION@server /usr/local/bundle/gems/debug-1.8.0/lib/debug/session.rb:179 sleep_forever> - <internal:thread_sync>:18:in `pop'
hellorails-cable-1     | [1] ! #<Rack::MiniProfiler::FileStore::CacheCleanupThread:0x00007f0df6324230 /usr/local/bundle/gems/rack-mini-profiler-3.1.1/lib/mini_profiler/storage/file_store.rb:67 sleep> - /usr/local/bundle/gems/rack-mini-profiler-3.1.1/lib/mini_profiler/storage/file_store.rb:85:in `sleep'
hellorails-cable-1     | [1] Use Ctrl-C to stop
hellorails-cable-1     | [1] ! WARNING: Detected running cluster mode with 1 worker.
hellorails-cable-1     | [1] ! Running Puma in cluster mode with a single worker is often a misconfiguration.
hellorails-cable-1     | [1] ! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead.
hellorails-cable-1     | [1] ! Set the `silence_single_worker_warning` option to silence this warning message.
hellorails-cable-1     | [1] - Worker 0 (PID: 11) booted in 0.0s, phase: 0
hellorails-worker-1    | Connection timed out - user specified timeout
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:65:in `connect_internal'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:141:in `connect'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:645:in `block in tcp'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:231:in `each'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:231:in `foreach'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:635:in `tcp'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client/ruby_connection.rb:117:in `connect'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client/ruby_connection.rb:49:in `initialize'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client.rb:684:in `new'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client.rb:684:in `block in connect'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client/middlewares.rb:12:in `connect'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client.rb:683:in `connect'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client.rb:670:in `raw_connection'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client.rb:637:in `ensure_connected'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client.rb:218:in `call'
hellorails-worker-1    | /usr/local/bundle/gems/redis-client-0.16.0/lib/redis_client/decorator.rb:26:in `call'
hellorails-worker-1    | /usr/local/bundle/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:145:in `block in redis_info'
hellorails-worker-1    | /usr/local/bundle/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:164:in `block in redis'
hellorails-worker-1    | /usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
hellorails-worker-1    | /usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
hellorails-worker-1    | /usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
hellorails-worker-1    | /usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
hellorails-worker-1    | /usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
hellorails-worker-1    | /usr/local/bundle/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:161:in `redis'
hellorails-worker-1    | /usr/local/bundle/gems/sidekiq-7.1.2/lib/sidekiq/config.rb:144:in `redis_info'
hellorails-worker-1    | /usr/local/bundle/gems/sidekiq-7.1.2/lib/sidekiq/cli.rb:75:in `run'
hellorails-worker-1    | /usr/local/bundle/gems/sidekiq-7.1.2/bin/sidekiq:31:in `<top (required)>'
hellorails-worker-1    | /usr/local/bundle/bin/sidekiq:25:in `load'
hellorails-worker-1    | /usr/local/bundle/bin/sidekiq:25:in `<top (required)>'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `load'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `kernel_load'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/cli.rb:492:in `exec'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/cli.rb:34:in `dispatch'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/cli.rb:28:in `start'
hellorails-worker-1    | /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.10/libexec/bundle:45:in `block in <top (required)>'
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
hellorails-worker-1    | /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.10/libexec/bundle:33:in `<top (required)>'
hellorails-worker-1    | /usr/local/bin/bundle:25:in `load'
hellorails-worker-1    | /usr/local/bin/bundle:25:in `<main>'
hellorails-worker-1 exited with code 1
hellorails-web-1       | Started GET "/" for 172.20.0.1 at 2023-09-13 15:02:45 +0000
hellorails-web-1       |   
hellorails-web-1       | ActiveRecord::ConnectionNotEstablished (connection to server at "172.20.0.4", port 5432 failed: Connection timed out
hellorails-web-1       |    Is the server running on that host and accepting TCP/IP connections?
hellorails-web-1       | ):
nickjj commented 9 months ago

Hi,

Did you modify anything in the .env file or database.yml config?

After upping the project what does docker compose ps show?

JeremiahChurch commented 9 months ago

Thanks for the reply!

Did you modify anything in the .env file or database.yml config?

no, sir - straight from the example - confirmed with another copy of the example just to be sure I didn't do something stupid (that's pretty common :)) here's a full fresh reboot of the host:

jchurch@ubuntu:~/hellorails$ cp .env.example .env
direnv: export +COMPOSE_PROFILES +COMPOSE_PROJECT_NAME +DOCKER_BUILDKIT +DOCKER_CABLE_PORT_FORWARD +DOCKER_RESTART_POLICY +DOCKER_WEB_HEALTHCHECK_TEST +DOCKER_WEB_PORT_FORWARD +DOCKER_WEB_VOLUME +NODE_ENV +POSTGRES_PASSWORD +POSTGRES_USER +RAILS_ENV +RAILS_MAX_THREADS +SECRET_KEY_BASE +WEB_CONCURRENCY
jchurch@ubuntu:~/hellorails$ docker compose up
[+] Running 7/0
 ✔ Container hellorails-redis-1     Created                                                                                                                                              0.0s 
 ✔ Container hellorails-postgres-1  Created                                                                                                                                              0.0s 
 ✔ Container hellorails-js-1        Created                                                                                                                                              0.0s 
 ✔ Container hellorails-cable-1     Created                                                                                                                                              0.0s 
 ✔ Container hellorails-web-1       Created                                                                                                                                              0.0s 
 ✔ Container hellorails-worker-1    Created                                                                                                                                              0.0s 
 ✔ Container hellorails-css-1       Created                                                                                                                                              0.0s 
Attaching to hellorails-cable-1, hellorails-css-1, hellorails-js-1, hellorails-postgres-1, hellorails-redis-1, hellorails-web-1, hellorails-worker-1
hellorails-redis-1     | 1:C 13 Sep 2023 16:10:25.890 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
hellorails-redis-1     | 1:C 13 Sep 2023 16:10:25.890 # Redis version=7.0.12, bits=64, commit=00000000, modified=0, pid=1, just started
hellorails-redis-1     | 1:C 13 Sep 2023 16:10:25.890 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.891 * Increased maximum number of open files to 10032 (it was originally set to 1024).
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.891 * monotonic clock: POSIX clock_gettime
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * Running mode=standalone, port=6379.
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 # Server initialized
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * Loading RDB produced by version 7.0.12
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * RDB age 48 seconds
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * RDB memory usage when created 0.82 Mb
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * Done loading RDB, keys loaded: 0, keys expired: 0.
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * DB loaded from disk: 0.000 seconds
hellorails-redis-1     | 1:M 13 Sep 2023 16:10:25.892 * Ready to accept connections
hellorails-postgres-1  | 
hellorails-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
hellorails-postgres-1  | 
hellorails-postgres-1  | 
hellorails-postgres-1  | 2023-09-13 16:10:26.040 UTC [1] LOG:  starting PostgreSQL 15.4 (Debian 15.4-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
hellorails-postgres-1  | 2023-09-13 16:10:26.041 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
hellorails-postgres-1  | 2023-09-13 16:10:26.041 UTC [1] LOG:  listening on IPv6 address "::", port 5432
hellorails-postgres-1  | 2023-09-13 16:10:26.042 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
hellorails-postgres-1  | 2023-09-13 16:10:26.045 UTC [29] LOG:  database system was shut down at 2023-09-13 16:09:37 UTC
hellorails-postgres-1  | 2023-09-13 16:10:26.056 UTC [1] LOG:  database system is ready to accept connections
yarn run v1.22.19

yarn run v1.22.19      | 
$ ./run yarn:build
$ ./run yarn:build:css
hellorails-js-1        | [watch] build finished, watching for changes...
hellorails-cable-1     | [1] Puma starting in cluster mode...
hellorails-cable-1     | [1] * Puma version: 6.3.1 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")
hellorails-cable-1     | [1] *  Min threads: 1
hellorails-cable-1     | [1] *  Max threads: 1
hellorails-cable-1     | [1] *  Environment: development
hellorails-cable-1     | [1] *   Master PID: 1
hellorails-cable-1     | [1] *      Workers: 1
hellorails-cable-1     | [1] *     Restarts: (✔) hot (✖) phased
hellorails-cable-1     | [1] * Preloading application
hellorails-css-1       | 
hellorails-css-1       | Rebuilding...
hellorails-css-1       | 
hellorails-css-1       | Done in 215ms.
hellorails-css-1       | 
hellorails-web-1       | => Booting Puma
hellorails-web-1       | => Rails 7.0.7.2 application starting in development 
hellorails-web-1       | => Run `bin/rails server --help` for more startup options
hellorails-web-1       | [1] Puma starting in cluster mode...
hellorails-web-1       | [1] * Puma version: 6.3.1 (ruby 3.2.2-p53) ("Mugi No Toki Itaru")
hellorails-web-1       | [1] *  Min threads: 1
hellorails-web-1       | [1] *  Max threads: 1
hellorails-web-1       | [1] *  Environment: development
hellorails-web-1       | [1] *   Master PID: 1
hellorails-web-1       | [1] *      Workers: 1
hellorails-web-1       | [1] *     Restarts: (✔) hot (✖) phased
hellorails-web-1       | [1] * Preloading application
hellorails-web-1       | [1] * Listening on http://0.0.0.0:8000
hellorails-web-1       | [1] Use Ctrl-C to stop
hellorails-web-1       | [1] ! WARNING: Detected running cluster mode with 1 worker.
hellorails-web-1       | [1] ! Running Puma in cluster mode with a single worker is often a misconfiguration.
hellorails-web-1       | [1] ! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead.
hellorails-web-1       | [1] ! Set the `silence_single_worker_warning` option to silence this warning message.
hellorails-web-1       | [1] - Worker 0 (PID: 11) booted in 0.0s, phase: 0
hellorails-worker-1    | 
hellorails-worker-1    | 
hellorails-worker-1    |                m,
hellorails-worker-1    |                `$b
hellorails-worker-1    |           .ss,  $$:         .,d$
hellorails-worker-1    |           `$$P,d$P'    .,md$P"'
hellorails-worker-1    |            ,$$$$$b/md$$$P^'
hellorails-worker-1    |          .d$$$$$$/$$$P'
hellorails-worker-1    |          $$^' `"/$$$'       ____  _     _      _    _
hellorails-worker-1    |          $:    ',$$:       / ___|(_) __| | ___| | _(_) __ _
hellorails-worker-1    |          `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
hellorails-worker-1    |                 $$:         ___) | | (_| |  __/   <| | (_| |
hellorails-worker-1    |                 $$         |____/|_|\__,_|\___|_|\_\_|\__, |
hellorails-worker-1    |               .d$$                                       |_|
hellorails-worker-1    |       
hellorails-worker-1    | 
hellorails-worker-1    | 2023-09-13T16:10:28.295Z pid=1 tid=2pt INFO: Booted Rails 7.0.7.2 application in development environment
hellorails-worker-1    | 2023-09-13T16:10:28.296Z pid=1 tid=2pt INFO: Running in ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]
hellorails-worker-1    | 2023-09-13T16:10:28.296Z pid=1 tid=2pt INFO: See LICENSE and the LGPL-3.0 for licensing details.
hellorails-worker-1    | 2023-09-13T16:10:28.296Z pid=1 tid=2pt INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
hellorails-worker-1    | 2023-09-13T16:10:28.296Z pid=1 tid=2pt INFO: Sidekiq 7.1.2 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://redis:6379/1"}
hellorails-cable-1     | [1] * Listening on http://0.0.0.0:28080
hellorails-cable-1     | [1] ! WARNING: Detected 2 Thread(s) started in app boot:
hellorails-cable-1     | [1] ! #<Thread:0x00007f722c1eb220@DEBUGGER__::SESSION@server /usr/local/bundle/gems/debug-1.8.0/lib/debug/session.rb:179 sleep_forever> - <internal:thread_sync>:18:in `pop'
hellorails-cable-1     | [1] ! #<Rack::MiniProfiler::FileStore::CacheCleanupThread:0x00007f722b3241d8 /usr/local/bundle/gems/rack-mini-profiler-3.1.1/lib/mini_profiler/storage/file_store.rb:67 sleep> - /usr/local/bundle/gems/rack-mini-profiler-3.1.1/lib/mini_profiler/storage/file_store.rb:85:in `sleep'
hellorails-cable-1     | [1] Use Ctrl-C to stop
hellorails-cable-1     | [1] ! WARNING: Detected running cluster mode with 1 worker.
hellorails-cable-1     | [1] ! Running Puma in cluster mode with a single worker is often a misconfiguration.
hellorails-cable-1     | [1] ! Consider running Puma in single-mode (workers = 0) in order to reduce memory overhead.
hellorails-cable-1     | [1] ! Set the `silence_single_worker_warning` option to silence this warning message.
hellorails-cable-1     | [1] - Worker 0 (PID: 11) booted in 0.0s, phase: 0
hellorails-worker-1    | Connection timed out - user specified timeout
hellorails-worker-1    | /usr/local/lib/ruby/3.2.0/socket.rb:65:in `connect_internal'
<trimmed stack>
hellorails-worker-1 exited with code 1
hellorails-web-1       | Started GET "/" for 172.20.0.1 at 2023-09-13 16:11:37 +0000
hellorails-web-1       |   
hellorails-web-1       | ActiveRecord::ConnectionNotEstablished (connection to server at "172.20.0.2", port 5432 failed: Connection timed out
hellorails-web-1       |    Is the server running on that host and accepting TCP/IP connections?
hellorails-web-1       | ):

After upping the project what does docker compose ps show?

jchurch@ubuntu:~/hellorails$ docker compose ps
NAME                    IMAGE                    COMMAND                  SERVICE             CREATED             STATUS                            PORTS
hellorails-cable-1      hellorails-cable         "puma -p 28080 cable…"   cable               About an hour ago   Up 9 seconds                      8000/tcp, 0.0.0.0:28080->28080/tcp, :::28080->28080/tcp
hellorails-css-1        hellorails-css           "yarn build:css"         css                 About an hour ago   Up 10 seconds                     
hellorails-js-1         hellorails-js            "yarn build"             js                  About an hour ago   Up 10 seconds                     
hellorails-postgres-1   postgres:15.4-bookworm   "docker-entrypoint.s…"   postgres            16 hours ago        Up 10 seconds                     5432/tcp
hellorails-redis-1      redis:7.0.12-bookworm    "docker-entrypoint.s…"   redis               16 hours ago        Up 10 seconds                     6379/tcp
hellorails-web-1        hellorails-web           "/app/bin/docker-ent…"   web                 About an hour ago   Up 9 seconds (health: starting)   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp
nickjj commented 9 months ago

If you up the project and in a 2nd terminal run run cmd curl https://example.com does it successfully connect to the internet and produce a response?

It looks like the app container can't connect over the local Docker network to the local postgres / redis containers but I'd be curious to see if the public network and DNS works in the container.

JeremiahChurch commented 9 months ago

seemingly no connections at all - internal or external (but still able to make requests to 8000 from the host to web)

curl to localhost does work (returns 500)

jchurch@ubuntu:~/hellorails$ ./run cmd curl https://example.com
curl: (28) Failed to connect to example.com port 443 after 130936 ms: Couldn't connect to server

Task completed in 2m11.105s
jchurch@ubuntu:~/hellorails$ ./run cmd curl http://localhost:8000
<!DOCTYPE html>
<removed- it's the rails console HTML page>

docker execs from the containers show the same thing

jchurch@ubuntu:~/hellorails$ docker exec -it hellorails-web-1 curl -w '\n' http://redis:6379/ping
curl: (28) Failed to connect to redis port 6379 after 129633 ms: Couldn't connect to server

definitely looks like a network comm issue. DNS resolution works because it does turn postgres in to the correct IP.

looks like enable_icc is default (good)

jchurch@ubuntu:~/hellorails$ docker inspect -f '{{index .Options "com.docker.network.bridge.enable_icc"}}' hellorails_default

That's the end of my docker networking knowledge

nickjj commented 9 months ago

Hmm, it does look like something on your Docker host is preventing networking from working inside of Docker.

How did you install Docker? Are you running through any proxy or VPN?

JeremiahChurch commented 9 months ago

no odd networking on the host - it's a ubuntu 20 LTS dev machine running on vmware workstation. Single wired nic with normal routered access to internet, no proxies, vpns, weird DNS, double NAT, etc.

docker install was via the published docker desktop instructions - https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository . I just did their standard 'complete removal' instructions followed by a fresh install, no luck there with a docker compose version of 2.21.0 now.

I've got a great direction to go - really appreciate your help! I'll follow up once I find the resolution but this certainly looks like a 'my machine' issue rather than a package issue

nickjj commented 9 months ago

Everything you mentioned looks normal.

On your Docker host can you temporarily modify your /etc/resolv.conf file to set your nameserver to8.8.8.8 or 1.1.1.1 to see if that makes a difference?

JeremiahChurch commented 9 months ago

On your Docker host can you temporarily modify your /etc/resolv.conf file to set your nameserver to8.8.8.8 or 1.1.1.1 to see if that makes a difference?

No luck there :\

I'm happy to call this a 'my machine' and not waste anymore of your time. I really appreciate your help!

I spun a second ubuntu 22 fresh VM, did the docker desktop install and fired the project up the first time with zero issues. docker network inspect was identical between both machines (minus UIDs.) 🤷

nickjj commented 9 months ago

I wonder what it was. If you end up figuring it out let us know by replying here.

nickjj commented 9 months ago

I wonder if it was related to iptables rules somehow. That would line up with networking not working.

There's some dragons hidden there around iptables-legacy vs the default although I only ever encountered that when running Debian / Ubuntu inside of WSL 2 where I needed to use iptables-legacy through sudo update-alternatives --config iptables. I never had to do this on a native Linux system with any distro version.

JeremiahChurch commented 9 months ago

@nickjj you were right there on the iptables! Looks like somehow I had some leftover snap install hanging around.

https://askubuntu.com/questions/1423293/ubuntu-22-04-docker-containers-not-accessible-from-outside got me working. specifically running: 

sudo iptables-legacy -P FORWARD ACCEPT
sudo chmod 666 /var/run/docker.sock