[BUG] Container starts, but mastodon doesn't

[Elbullazul]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Once an unexpected container shutdown happens (ex. power loss of the server), the container will start up again, but mastodon won't, even after waiting a long time.

I found some errors in /config/log/nginx/error.log, but they don't seem related?

2024/04/28 09:28:31 [error] 286#286: *34362 connect() failed (111: Connection refused) while connecting to upstream, client: 172.16.25.4, server: _, request: "POST /inbox HTTP/1.1", upstream: "http://[::1]:3000/inbox", host: "<mastodon.domain>"

Expected Behavior

The container should be able to recover after an unexpected shutdown.

Steps To Reproduce

1. Create a mastodon container, it works fine
2. Reboot the server
3. The container will start, but never get past using keys found in /config/keys

Environment

- OS: Debian 12
- How docker service was installed: Docker's official script

CPU architecture

x86-64

Docker creation

---
version: "2.1"

services:
  mastodon:
    image: lscr.io/linuxserver/mastodon:latest
    container_name: mastodon
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=$TZ
      - LOCAL_DOMAIN=$DOMAIN
      - REDIS_HOST=mastodon-redis
      - REDIS_PORT=6379
      - DB_HOST=mastodon-db
      - DB_USER=$DB_USER
      - DB_NAME=$DB_NAME
      - DB_PASS=$DB_PASS
      - DB_PORT=5432
      - ES_ENABLED=false
      - SECRET_KEY_BASE=$SESSION_SECRET
      - OTP_SECRET=$OTP_SECRET
      - VAPID_PRIVATE_KEY=$VAPID_PRIVATE_KEY
      - VAPID_PUBLIC_KEY=$VAPID_PUBLIC_KEY
      - SMTP_SERVER=
      - SMTP_PORT=
      - SMTP_LOGIN=
      - SMTP_PASSWORD=
      - SMTP_FROM_ADDRESS=
      - SMTP_SSL=true
      - S3_ENABLED=false
      - DB_POOL=5 #optional
      - RAILS_LOG_LEVEL=info  # info for debug, warn in normal operation
      - LDAP_ENABLED=true
      - LDAP_METHOD=plain
      - LDAP_HOST=lldap
      - LDAP_PORT=3890
      - LDAP_BASE=dc=
      - LDAP_BIND_DN=
      - LDAP_PASSWORD=$LDAP_PASS
      - LDAP_UID=uid
      - LDAP_SEARCH_FILTER=
      - LDAP_MAIL=mail
      - LDAP_UID_CONVERSION_ENABLED=true
      - STATSD_ADDR=localhost:9125      # for prometheus (not working?)
    volumes:
      - ${BINDS}/mastodon/config:/config
    dns:
      - 1.1.1.1    # cloudflare
    depends_on:
      - mastodon-db
      - mastodon-redis

    restart: unless-stopped
    networks:
      - backend_mastodon
      - shn_mastodon
      - lnk_prometheus  # for metrics
    labels:
      com.centurylinklabs.watchtower.enable: true
      com.centurylinklabs.watchtower.depends-on: "/mastodon-db, /mastodon-redis"

  mastodon-db:
    image: postgres:15 #latest
    container_name: mastodon-db
    environment:
      - POSTGRES_PASSWORD=$DB_PASS
      - POSTGRES_USER=$DB_USER
      - POSTGRES_DB=$DB_NAME
    volumes:
      - $BINDS/mastodon/pgdb:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d ${DB_NAME} -U ${DB_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped
    networks:
      - backend_mastodon
      - shn_backends
    labels:
      com.centurylinklabs.watchtower.enable: true

  mastodon-redis:
    image: docker.io/redis:latest
    container_name: mastodon-redis
    volumes:
      - ${BINDS}/mastodon/cache:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 60s
      timeout: 20s
      retries: 5
    restart: unless-stopped
    networks:
      - backend_mastodon
    labels:
      com.centurylinklabs.watchtower.enable: true

networks:
  lnk_prometheus:
    external: true

  backend_mastodon:
    external: true
  shn_backends:
    external: true
  shn_mastodon:
    external: true

Container logs

[migrations] started
[migrations] 01-nginx-site-confs-default: skipped
[migrations] done
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1000
User GID:    1000
───────────────────────────────────────

using keys found in /config/keys

[github-actions[bot]]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[thespad]

If the container isn't progressing past that point then something is fundamentally broken as that's before any of the mastodon services try to start or any attempt to connect to databases/redis/etc are made.

The only possible issue is related to a huge /config mount and a poor-performing filesystem, in which case you can try setting NO_CHOWN=true to skip the permissions checks and see if that allows it to start.

[Elbullazul]

thanks for the suggestion, I'll test this when I can and report back if it works

[Elbullazul]

it does indeed allow the container to start faster. Is there any downside of using this option?

[thespad]

Only that you have to keep your own permissions in order. Not generally an issue but some platforms like Unraid have a habit of messing with permissions on the host side, which is why we chown on startup by default.

It will eventually start regardless, but if you've got a big server and/or not great storage, it can take upwards of 10 minutes to complete the process, even if it doesn't have to actually change any permissions because it still has to walk the whole /config mount to check.

[Elbullazul]

gotcha, so I shouldn't have any problems using vanilla debian + docker?

[thespad]

I wouldn't expect to, no; the only caveat is when setting up a new instance you need to make sure NO_CHOWN=true isn't set on first run.

[OutOfThisPlanet]

It will eventually start regardless, but if you've got a big server and/or not great storage, it can take upwards of 10 minutes to complete the process, even if it doesn't have to actually change any permissions because it still has to walk the whole /config mount to check.

This was an important piece of information for me, thanks!