Redis exception on boot - Client sent AUTH, but no password is set

[Rinma]

Hi,

when I try to start with docker-compose I get this error

zulip_1      | === Begin Bootstrap Phase ===
zulip_1      | Waiting for database server to allow connections ...
zulip_1      | Executing Zulip first start init ...
zulip_1      | +++ readlink -f /home/zulip/deployments/current/scripts/setup/initialize-database
zulip_1      | ++ dirname /home/zulip/deployments/2020-01-17-00-55-51/scripts/setup/initialize-database
zulip_1      | + THIS_DIR=/home/zulip/deployments/2020-01-17-00-55-51/scripts/setup
zulip_1      | + cd /home/zulip/deployments/2020-01-17-00-55-51/scripts/setup/../..
zulip_1      | + ./manage.py checkconfig
zulip_1      | + ./manage.py migrate --noinput
zulip_1      | Operations to perform:
zulip_1      |   Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
zulip_1      | Running migrations:
zulip_1      |   Applying zerver.0260_missed_message_addresses_from_redis_to_db...Traceback (most recent call last):
zulip_1      |   File "./manage.py", line 46, in <module>
zulip_1      |     execute_from_command_line(sys.argv)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
zulip_1      |     utility.execute()
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
zulip_1      |     self.fetch_command(subcommand).run_from_argv(self.argv)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
zulip_1      |     self.execute(*args, **cmd_options)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
zulip_1      |     output = self.handle(*args, **options)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 204, in handle
zulip_1      |     fake_initial=fake_initial,
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 115, in migrate
zulip_1      |     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
zulip_1      |     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
zulip_1      |     state = migration.apply(state, schema_editor)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 129, in apply
zulip_1      |     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
zulip_1      |     self.code(from_state.apps, schema_editor)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zerver/migrations/0260_missed_message_addresses_from_redis_to_db.py", line 23, in move_missed_message_addresses_to_database
zulip_1      |     all_mm_keys = redis_client.keys('missed_message:*')
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/client.py", line 1387, in keys
zulip_1      |     return self.execute_command('KEYS', pattern)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/client.py", line 836, in execute_command
zulip_1      |     conn = self.connection or pool.get_connection(command_name, **options)
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/connection.py", line 1073, in get_connection
zulip_1      |     connection.connect()
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/connection.py", line 548, in connect
zulip_1      |     self.on_connect()
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/connection.py", line 618, in on_connect
zulip_1      |     if nativestr(self.read_response()) != 'OK':
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/connection.py", line 700, in read_response
zulip_1      |     response = self._parser.read_response()
zulip_1      |   File "/home/zulip/deployments/2020-01-17-00-55-51/zulip-py3-venv/lib/python3.6/site-packages/redis/connection.py", line 327, in read_response
zulip_1      |     raise error
zulip_1      | redis.exceptions.AuthenticationError: Client sent AUTH, but no password is set
zulip_1      | Zulip first start database initi failed in "initialize-database" exit code 1. Exiting.

my docker-compose file:

version: '2'
services:
  database:
    image: 'zulip/zulip-postgresql:10'
    environment:
      POSTGRES_DB: 'zulip'
      POSTGRES_USER: 'zulip'
      # Note that you need to do a manual `ALTER ROLE` query if you
      # change this on a system after booting the postgres container
      # the first time on a host.  Instructions are available in README.md.
      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
    volumes:
      - 'zulip-database:/var/lib/postgresql/data:rw'
    networks:
      - zulip
  memcached:
    image: 'memcached:alpine'
    command:
      - 'sh'
      - '-euc'
      - |
        echo 'mech_list: plain' > "$$SASL_CONF_PATH"
        echo "zulip@$$HOSTNAME:$$MEMCACHED_PASSWORD" > "$$MEMCACHED_SASL_PWDB"
        exec memcached -S
    environment:
      SASL_CONF_PATH: '/home/memcache/memcached.conf'
      MEMCACHED_SASL_PWDB: '/home/memcache/memcached-sasl-db'
      MEMCACHED_PASSWORD: ${CACHE_PASSWORD}
    restart: always
    networks:
      - zulip
  rabbitmq:
    image: 'rabbitmq:3.7.7'
    hostname: zulip-rabbit
    restart: always
    environment:
        RABBITMQ_DEFAULT_USER: 'zulip'
        RABBITMQ_DEFAULT_PASS: ${MESSAGE_QUEUE_PASSWORD}
    volumes:
      - 'zulip-messages:/var/lib/rabbitmq:rw'
    networks:
      - zulip
  redis:
    image: 'redis:alpine'
    command:
      - 'sh'
      - '-euc'
      - |
        echo "requirepass '$$REDIS_PASSWORD'" > /etc/redis.conf
        exec redis-server /etc/redis.conf
    environment:
      REDIS_PASSWORD: ${EXTERNAL_REDIS_PASSWORD}
    volumes:
      - 'zulip-redis:/data:rw'
    networks:
      - zulip
  zulip:
    image: 'zulip/docker-zulip:2.1.2-0'
    build:
      context: .
      args:
        # Change these if you want to build zulip from a different repo/branch
        ZULIP_GIT_URL: https://github.com/zulip/zulip.git
        ZULIP_GIT_REF: 2.1.2
        # Set this up if you plan to use your own CA certificate bundle for building
        # CUSTOM_CA_CERTIFICATES:
#    ports:
#      - '80:80'
#      - '443:443'
    restart: unless-stopped
    depends_on:
    - database
    - memcached
    - rabbitmq
    - redis
    labels:
    - "traefik.enable=true"
    - "traefik.network=traefik"

    - "traefik.http.routers.zulip_router_insecure.rule=Host(`chat.example.org`)"
    - "traefik.http.routers.zulip_router_insecure.entrypoints=web"
    - "traefik.http.routers.zulip_router_insecure.middlewares=zulip-redirect-to-https"

    - "traefik.http.routers.zulip_router.rule=Host(`chat.example.org`)"
    - "traefik.http.routers.zulip_router.entrypoints=secure"
    - "traefik.http.routers.zulip_router.tls=true"
    - "traefik.http.routers.zulip_router.tls.certresolver=letsencrypt"

    - "traefik.http.middlewares.zulip-redirect-to-https.redirectscheme.scheme=https"
    environment:
      DB_HOST: 'database'
      DB_HOST_PORT: '5432'
      DB_USER: 'zulip'
      SSL_CERTIFICATE_GENERATION: 'self-signed'
      SETTING_MEMCACHED_LOCATION: 'memcached:11211'
      SETTING_RABBITMQ_HOST: 'rabbitmq'
      SETTING_REDIS_HOST: 'redis'
      SECRETS_email_password: ${SMTP_PASSWORD}
      # These should match RABBITMQ_DEFAULT_PASS, POSTGRES_PASSWORD,
      # MEMCACHED_PASSWORD, and REDIS_PASSWORD above.
      SECRETS_rabbitmq_password: ${MESSAGE_QUEUE_PASSWORD}
      SECRETS_postgres_password: ${DATABASE_PASSWORD}
      SECRETS_memcached_password: ${CACHE_PASSWORD}
      SECRETS_redis_password: ${EXTERNAL_REDIS_PASSWORD}
      SECRETS_secret_key: ${SECRET_KEY}
      SETTING_EXTERNAL_HOST: ${DOMAIN}
      SETTING_ZULIP_ADMINISTRATOR: 'admin@example.org'
      SETTING_EMAIL_HOST: ${SMTP_HOST}
      SETTING_EMAIL_HOST_USER: ${SMTP_USER}
      SETTING_EMAIL_PORT: '587'
      # It seems that the email server needs to use ssl or tls and can't be used without it
      SETTING_EMAIL_USE_SSL: 'False'
      SETTING_EMAIL_USE_TLS: 'True'
      ZULIP_AUTH_BACKENDS: 'EmailAuthBackend'
      # Uncomment this when configuring the mobile push notifications service
      # SETTING_PUSH_NOTIFICATION_BOUNCER_URL: 'https://push.zulipchat.com'
    volumes:
      - 'zulip-data:/data:rw'
    ulimits:
      nofile:
        soft: 40000
        hard: 50000
    networks:
      - zulip
      - traefik

volumes:
    zulip-data:
    zulip-database:
    zulip-messages:
    zulip-redis:

networks:
  zulip:
    driver: bridge
  traefik:
    external: true

The passwords are in a .env file in the root of the project where the docker-compose file is.

Any ideas what the problem is?

[timabbott]

Did you upgrade from an older version of Zulip?

@andersk FYI as this is in migration code using redis.

[Rinma]

@timabbott No new installation. Zulip was never started before on this server.

[blubbi321]

Same issue here with an upgrade from 2.1.1-0 => 2.1.2-0

(Also I noted that the docker-compose.yml has changed quite a bit between the releases .. maybe we need to include instructions for what to do about that during an upgrade in the readme?)

[timabbott]

@blubbi321 well for the 2.1.1 -> 2.1.2 upgrade, I imagine at least you'll need to set the new redis and memcached password settings -- that's what I was expecting.

I'm less sure what the story is for someone with a new installation.

[blubbi321]

@timabbott Ok I see. But given https://github.com/zulip/docker-zulip/commit/f2ae767f9a8f3722f20a951c2a387abb4e6f78af I can see that the path to redis changes and hence I have doubts that the upgrade instructions as in here https://github.com/zulip/docker-zulip#using-docker-compose will work. In fact I think they broke my installation now ;)

If all I need to do is to update the docker-compose.yml with the diff from above and set the password that would be great but Ill rather wait for your confirmation before things get more broken.

[ilanpillemer]

well I added the REDIS PASSWORD, which got it past that for a new installation. But now it hanging on

2020-02-04 23:41:03,663 INFO success: zulip_events_error_reports entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-04 23:41:03,663 INFO success: zulip_events_digest_emails entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-04 23:41:03,663 INFO success: zulip_events_email_senders entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-04 23:41:03,663 INFO success: zulip_deliver_scheduled_messages entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

At least I think its hanging.. been like that for a while?

[ilanpillemer]

btw, I am using the k8s files in the repo and trying on k8s.. where its hanging now on this. (after I fixed the REDIS error by adding the env variable.)

[ilanpillemer]

ok. it hanged in that state all night.. any suggestions on how to proceed?

[xmaelo]

ok thank