Error accessing Zulip secrets

[Lionelpang]

hi, Start the service through docker-compose and report the error： Error accessing Zulip secrets; manage.py in production must be run as the zulip user. What can I do to solve this problem?

[timabbott]

@Lionelpang can you give more detail on exactly what you're running, with more terminal history? I cannot reproduce; this Docker should always correctly run manage.py as the zulip user.

[timabbott]

Closing, since I can't reproduce and there's no information here we can use the debug further. @Lionelpang if you can provide more detail, we can reopen this issue.

[timabbott]

There was a similar report brought up here: https://chat.zulip.org/#narrow/stream/2-general/subject/docker-zulip/near/660861, we're debugging there.

[Gaulomatic]

I am the error as well and the link which you provided is not accessable. Nothing special configured, so essentially git clone, docker-compose up.

[timabbott]

@Gaulomatic can you talk about what platform you're on? I have a theory that things might work if you try the version of the code in #166, but would like confirmation.

[Gaulomatic]

Ok, I was able to solve the problem; but I don't know really why. It is definitely a permission issue. I mapped the /data dir to something on the host that had extended ACLs on it (necessary for some nasty backup stuff). I got the error. Then I mapped it outside this directory structure and it worked. Then I copied the files over to the ACL contaminated location, got the error again. Then I gave others at least read permission (644/755) -> worked. The error message is misleading, but I guess this is an "edge case" (extended ACLs on Linux feels like the most edgy case in history of operating systems).

My system is an Ubuntu 18.04 box.

I took a look at the modified docker-compose.yml. What exactly is the benefit of using these placeholders? I could try this out, but I doubt this would change something since - in my case - it is somehow permissions related, something screws the effective permissions. Something is different than Zulip expect it to be.

[timabbott]

The benefit is that if you're using the Docker snap package (at least as @andersk was testing yesterday), one can't write to /opt/docker, but can write to the path that's generated via that volumes configuration.

[BNolet]

@Gaulomatic using named volumes would have resulted in permissions on the host not being an issue as the volumes would be written to a different directory (namely /var/lib/docker/volumes/) which docker and the docker group has write permissions on.

Another benefit is that since all the volumes are in the same directory, migration is a little simpler.

[timabbott]

Based on https://chat.zulip.org/#narrow/stream/2-general/subject/docker-zulip/near/677493, it seems very old versions of docker-compose (e.g. 1.6.0) caused this error for some users. Upgrading to 1.23.1 fixed the issue for the user. The traceback was this:

[ClundXIII]

I am getting the same error. I guess this is because I am using a custom .py and secrets config. I already tried tinkering with the settings and set the following perms:

# ll /data/docker/zulip/zulip/
total 84
drwxr-xr-x 2 root  root   4096 Nov 20 23:24 backups
drwxr-xr-x 2 root  root   4096 Nov 20 23:29 certs
drwxr-xr-x 2 root  root   4096 Nov 20 23:24 letsencrypt
-rwxr-xr-x 1 root  root  18347 Nov 20 23:22 prod_settings.py
drwxr-xr-x 3 root  root   4096 Nov 20 23:24 settings
-rwxr-xr-x 1 simon simon 40267 Nov 20 23:00 settings.py
drwxr-xr-x 2 simon simon  4096 Nov 20 23:24 uploads
-rw-r--r-- 1 simon simon   258 Nov 20 23:22 zulip-secrets.conf

I guess the user id of simon on host is the userid of zulip within the container (it was set by the installer on the upload directory).

Any idea how to fix this? I am running debian10 buster. Or do you suggest that I switch to the other newer version proposed in the PR from last year since I am doing a fresh setup?

[ClundXIII]

Okay I figured it out. The error message is misleading and caused by this piece of code:

    HAS_SECRETS = os.access('/etc/zulip/zulip-secrets.conf', os.R_OK)

    if PRODUCTION and not HAS_SECRETS:
        # The best way to detect running manage.py as another user in
        # production before importing anything that would require that
        # access is to check for access to /etc/zulip/zulip.conf (in
        # which case it's a production server, not a dev environment)
        # and lack of access for /etc/zulip/zulip-secrets.conf (which
        # should be only readable by root and zulip)
        print("Error accessing Zulip secrets; manage.py in production must be run as the zulip user.")
        sys.exit(1)

The config file is not properly deployed as described in the documentation.

[zodiac7]

I am also facing this issue - https://github.com/zulip/docker-zulip/issues/151 I am trying to use zulip in docker mode by using docker zulip with my own settings and secret files as described in https://github.com/zulip/docker-zulip#manual-configuration

I am using the default code from above but only editing the docker-compose.yml and adding the line MANUAL_CONFIGURATION: "True"

My Settings file and secret files are present in host at /opt/docker/zulip/zulip/settings/etc-zulip/, Root user owns them. Question - IS zulip.conf also required in this directory?

Zulip does not start and in the console says "Zulip migration failed with exit code 1". Details from log below. If i comment out MANUAL_CONFIGURATION in docker-compose.yml then everything runs file. I am running the latest docker-zulip cloned from github on Linux mint cinnamon 20.02 , docker compose 1.25.0-1

Please advise.

zulip_1 | Migrating Zulip to new version ... zulip_1 | Traceback (most recent call last): zulip_1 | File "/home/zulip/deployments/current/manage.py", line 46, in zulip_1 | log_management_command(sys.argv, settings.MANAGEMENT_LOG_PATH) zulip_1 | File "/srv/zulip-venv-cache/a7fb0fe17c8b91b591927bfa8f133a90b20f3d18/zulip-py3-venv/lib/python3.8/site-packages/django/conf/init.py", line 82, in getattr zulip_1 | self._setup(name) zulip_1 | File "/srv/zulip-venv-cache/a7fb0fe17c8b91b591927bfa8f133a90b20f3d18/zulip-py3-venv/lib/python3.8/site-packages/django/conf/init.py", line 69, in _setup zulip_1 | self._wrapped = Settings(settings_module) zulip_1 | File "/srv/zulip-venv-cache/a7fb0fe17c8b91b591927bfa8f133a90b20f3d18/zulip-py3-venv/lib/python3.8/site-packages/django/conf/init.py", line 170, in init zulip_1 | mod = importlib.import_module(self.SETTINGS_MODULE) zulip_1 | File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module zulip_1 | return _bootstrap._gcd_import(name[level:], package, level) zulip_1 | File "", line 1014, in _gcd_import zulip_1 | File "", line 991, in _find_and_load zulip_1 | File "", line 975, in _find_and_load_unlocked zulip_1 | File "", line 671, in _load_unlocked zulip_1 | File "", line 848, in exec_module zulip_1 | File "", line 219, in _call_with_frames_removed zulip_1 | File "/home/zulip/deployments/2022-02-24-13-32-57/zproject/settings.py", line 19, in zulip_1 | from .configured_settings import # noqa: F401,F403 isort: skip zulip_1 | File "/home/zulip/deployments/2022-02-24-13-32-57/zproject/configured_settings.py", line 8, in zulip_1 | from .default_settings import # noqa: F401,F403 isort: skip zulip_1 | File "/home/zulip/deployments/2022-02-24-13-32-57/zproject/default_settings.py", line 15, in zulip_1 | from .dev_settings import EXTERNAL_HOST, ZULIP_ADMINISTRATOR zulip_1 | ModuleNotFoundError: No module named 'zproject.dev_settings' zulip_1 | Zulip migration failed with exit code 1. Exiting. docker-zulip_zulip_1 exited with code 1

[zodiac7]

FYI @benjaminadamson and anyone who is facing this issue, I was able to make progress by COPYING all files from the container's etc directory to your host machine at /opt/docker/zulip/zulip/settings/etc-zulip. The files which I copied were nginx_sharding.conf, sharding.json, zulip.conf, settings.py, uwsgi.ini, zulip-secrets.conf

@timabbott I suggest updating the documentation for docker zulip with this information so others dont face this. Let me know if you would like me to update the docs and I will be happy to do it.

[n0099]

For anyone using userns-remap and/or rootless for docker like https://github.com/zulip/docker-zulip/issues/254:

1. assume the sub(u|g)id for the default user dockremap being used by userns-remap is 558752:

   tail -n +1 /etc/sub*id

   
   ==> /etc/subgid <==
   dockremap:558752:65536

==> /etc/subuid <== dockremap:558752:65536

YMMV so replace it with your own variant
2. assume your docker compose [project name](https://stackoverflow.com/questions/62114797/what-is-the-purpose-of-p-project-name-option-in-docker-compose-command) is `zulip`
3. following https://stackoverflow.com/questions/73175827/chown-breaks-in-bound-volumes-with-docker-user-namespace-remapping-operation-n, run `chown -R root:` as `558752` is the starting point `uid0` of ``sub(u|g)id`` range
```bash
sudo chown -R 558752:558752 /var/lib/docker/558752.558752/volumes/zulip_{postgresql-14,rabbitmq,redis,zulip}/_data

4. chown zulip:root as plus 558752 by the uid1000 of user zulip in container zulip_zulip

   sudo chown 559752:558752 /var/lib/docker/558752.558752/volumes/zulip_zulip/_data/zulip-secrets.conf

[n0099]

docker exec -it zulip-zulip-1 cat /etc/passwd

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
_apt:x:42:65534::/nonexistent:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash
puppet:x:100:101:Puppet configuration management daemon,,,:/var/lib/puppet:/usr/sbin/nologin
zulip:x:1001:1001::/home/zulip:/bin/bash
messagebus:x:101:102::/nonexistent:/usr/sbin/nologin
_chrony:x:102:103:Chrony daemon,,,:/var/lib/chrony:/usr/sbin/nologin
systemd-network:x:998:998:systemd Network Management:/:/usr/sbin/nologin
systemd-resolve:x:996:996:systemd Resolver:/:/usr/sbin/nologin

It seems the uid of user zulip changed from 1000 to 1001:

sudo chown 559753:558752 /var/lib/docker/558752.558752/volumes/zulip_zulip/_data/zulip-secrets.conf