Cannot update site URL in docker deployment

Describe the bug

In a docker-based Weblate installation, I initially used the server IP address in WEBLATE_ALLOWED_HOSTS. As part of productionizing the setup, I changed it to an actual URL weblate.lat.sk. However, the installation keeps insisting on using the IP address and reverts back on restart.

To Reproduce

Steps to reproduce the behavior:

Install Weblate using docker with WEBLATE_ALLOWED_HOSTS set to an IP address
Start Weblate
Update WEBLATE_ALLOWED_HOSTS to an URL
Update Site in Django admin / Sites to an URL
Run docker-compose restart

Expected behavior

The site is set to the URL

Server configuration and status

Weblate: 4.1.1
Django: 3.0.7
siphashc: 1.3
Whoosh: 2.7.4
translate-toolkit: 3.0.0
lxml: 4.5.1
Pillow: 7.1.2
bleach: 3.1.5
python-dateutil: 2.8.1
social-auth-core: 3.3.3
social-auth-app-django: 3.4.0
django-crispy-forms: 1.9.1
oauthlib: 3.1.0
django-compressor: 2.4
djangorestframework: 3.11.0
django-filter: 2.3.0
django-appconf: 1.0.4
user-agents: 2.1
filelock: 3.0.12
setuptools: 40.8.0
jellyfish: 0.8.2
openpyxl: 3.0.1
celery: 4.4.5
kombu: 4.6.10
translation-finder: 2.1
html2text: 2020.1.16
pycairo: 1.16.2
pygobject: 3.30.4
diff-match-patch: 20181111
requests: 2.24.0
django-redis: 4.12.1
hiredis: 1.0.1
sentry_sdk: 0.15.1
Cython: 0.29.20
misaka: 2.1.1
GitPython: 3.1.3
borgbackup: 1.1.13
pyparsing: 2.4.7
Python: 3.7.3
Git: 2.20.1
psycopg2: 2.8.5
psycopg2-binary: 2.8.5
phply: 1.2.5
chardet: 3.0.4
ruamel.yaml: 0.16.10
tesserocr: 2.5.1
akismet: 1.1
boto3: 1.14.6
zeep: 3.4.0
aeidon: 1.7.0
iniparse: 0.5
mysqlclient: 1.4.6
Mercurial: 5.4.1
git-svn: 2.20.1
git-review: 1.28.0
hub: 2.13.0
lab: 0.16
Redis server: 5.0.9
PostgreSQL server: 11.8
Database backends: django.db.backends.postgresql
Cache backends: default:RedisCache, avatar:FileBasedCache
Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.gmail.com
OS encoding: filesystem=utf-8, default=utf-8
Celery: redis://cache:6379/1, redis://cache:6379/1, regular
Platform: Linux 5.4.0-40-generic (x86_64)

Weblate deploy checks

System check identified some issues:

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
        HINT: https://docs.weblate.org/en/weblate-4.1.1/admin/backup.html

System check identified 2 issues (1 silenced).

Exception traceback

database_1  | 2020-07-05 11:18:59.794 UTC [67] ERROR:  duplicate key value violates unique constraint "django_site_domain_a2e37b91_uniq"
database_1  | 2020-07-05 11:18:59.794 UTC [67] DETAIL:  Key (domain)=(weblate.lat.sk) already exists.
database_1  | 2020-07-05 11:18:59.794 UTC [67] STATEMENT:  UPDATE "django_site" SET "domain" = 'weblate.lat.sk', "name" = 'weblate.lat.sk' WHERE "django_site"."id" = 1

However, when app containers start, I login to the database and run the same update statement, it passes without errors.

Additional context

When I set the site URL via Django Admin / Sites, it gets reverted back to the IP address on docker restart. When I set the site URL via docker-compose exec weblate weblate changesite --set-name weblate.lat.sk, it gets reverted as well on restart.

Additionally, I have no idea where Weblate gets the IP address from as it is not present in any configuration file.

docker-compose.override.yml

version: '3'
services:
  weblate:
    ports:
      - 8182:8080
      - 8183:4443
    environment:
      WEBLATE_EMAIL_HOST: smtp.***.com
      WEBLATE_EMAIL_HOST_USER: ***@***.com
      WEBLATE_EMAIL_HOST_PASSWORD: ***
      WEBLATE_EMAIL_PORT: ***
      WEBLATE_EMAIL_USE_TLS: 1
      WEBLATE_SERVER_EMAIL: ***@***.com
      WEBLATE_DEFAULT_FROM_EMAIL: ***@***.com
      WEBLATE_ALLOWED_HOSTS: weblate.lat.sk
      WEBLATE_ADMIN_PASSWORD: ***
      WEBLATE_ADMIN_EMAIL: ***@***.com
      WEBLATE_DEBUG: 0
      WEBLATE_SECURE_PROXY_SSL_HEADER: HTTP_X_FORWARDED_PROTO,https
      WEBLATE_ENABLE_HTTPS: 1

settings-override.py

In: /var/lib/docker/volumes/weblate-docker_weblate-data/_data/settings-override.py

MT_SERVICES = (
#    'weblate.machinery.apertium.ApertiumAPYTranslation',
#    'weblate.machinery.deepl.DeepLTranslation',
#    'weblate.machinery.glosbe.GlosbeTranslation',
    'weblate.machinery.google.GoogleTranslation',
#    'weblate.machinery.microsoft.MicrosoftCognitiveTranslation',
#    'weblate.machinery.microsoftterminology.MicrosoftTerminologyService',
#    'weblate.machinery.mymemory.MyMemoryTranslation',
#    'weblate.machinery.tmserver.AmagamaTranslation',
#    'weblate.machinery.tmserver.TMServerTranslation',
#    'weblate.machinery.yandex.YandexTranslation',
#    'weblate.machinery.weblatetm.WeblateTranslation',
#    'weblate.machinery.saptranslationhub.SAPTranslationHub',
#    'weblate.memory.machine.WeblateMemory',
)

MT_GOOGLE_KEY="***"

SECURE_SSL_REDIRECT=True

SECURE_HSTS_PRELOAD=True
SECURE_HSTS_INCLUDE_SUBDOMAINS=True
SECURE_HSTS_SECONDS=63072000

SESSION_COOKIE_SECURE=True

SENTRY_DNS="https://***@***.ingest.sentry.io/***"

You probably manually added another entry to Django sites database with same domain. Go to that admin interface, delete it and fix the one with ID 1 (or restart the container).

I checked both the Django admin interface for Sites and the django_site database table. There is only one record with the IP address and ID 1.

I tried to debug this issue a little more. I noticed the site name doesn't get reset to the IP address immediately after restart. So I simultaneously watched that table and weblate log to correlate the change. However, there doesn't seem to be anything evidently related. Maybe you will have an idea.

Is there anything else I can try to debug this issue? So far, the only workaround it to manually set it again after every restart.

Database change

2020-07-05 19:52:38.384724+00 | weblate.lat.sk
2020-07-05 19:52:38.403497+00 | <IP ADDRESS>

Weblate log

weblate_1   | Deleting 'font-source/OTF/SourceCodePro-Regular.otf'
weblate_1   | Deleting 'font-source/OTF/SourceCodePro-Bold.otf'
weblate_1   | Deleting 'font-source/OTF/SourceSansPro-Regular.otf'
weblate_1   | Deleting 'font-source/OTF/SourceSansPro-Black.otf'
weblate_1   | Deleting 'font-dejavu/DejaVuSans-Bold.ttf'
weblate_1   | Deleting 'font-dejavu/DejaVuSans.ttf'
weblate_1   | Deleting 'font-dejavu/.uuid'
weblate_1   |
weblate_1   | 382 static files copied to '/app/data/static'.
weblate_1   | 2020-07-05 19:52:28,836 DEBUG: git: exec git --version [retcode=0]
weblate_1   | 2020-07-05 19:52:29,611 DEBUG: git: exec git review --version [retcode=0]
weblate_1   | 2020-07-05 19:52:30,088 DEBUG: git: exec git svn --version [retcode=0]
weblate_1   | 2020-07-05 19:52:30,116 DEBUG: hub: exec hub --version [retcode=0]
weblate_1   | 2020-07-05 19:52:30,156 DEBUG: lab: exec lab --version [retcode=1]
weblate_1   | 2020-07-05 19:52:30,550 DEBUG: hg: exec hg version -q [retcode=0]
weblate_1   | 2020-07-05 19:52:31,448 DEBUG: Attempting to acquire lock 140396378918808 on /app/data/home/gitlock
weblate_1   | 2020-07-05 19:52:31,450 INFO: Lock 140396378918808 acquired on /app/data/home/gitlock
weblate_1   | 2020-07-05 19:52:31,467 DEBUG: git: exec git config --global merge.weblate-merge-gettext-po.name Weblate merge driver for Gettext PO files [retcode=0]
weblate_1   | 2020-07-05 19:52:31,481 DEBUG: git: exec git config --global merge.weblate-merge-gettext-po.driver /usr/local/lib/python3.7/dist-packages/weblate/examples/git-merge-gettext-po %O %A %B [retcode=0]
weblate_1   | 2020-07-05 19:52:31,497 DEBUG: git: exec git config --global user.email noreply@weblate.org [retcode=0]
weblate_1   | 2020-07-05 19:52:31,512 DEBUG: git: exec git config --global user.name Weblate [retcode=0]
weblate_1   | 2020-07-05 19:52:31,528 DEBUG: Using selector: EpollSelector
weblate_1   | 2020-07-05 19:52:31,550 DEBUG: Attempting to release lock 140396378918808 on /app/data/home/gitlock
weblate_1   | 2020-07-05 19:52:31,551 INFO: Lock 140396378918808 released on /app/data/home/gitlock
weblate_1   | Updating user admin
weblate_1   | 2020-07-05 19:52:35,591 DEBUG: git: exec git --version [retcode=0]
weblate_1   | 2020-07-05 19:52:36,358 DEBUG: git: exec git review --version [retcode=0]
weblate_1   | 2020-07-05 19:52:36,847 DEBUG: git: exec git svn --version [retcode=0]
weblate_1   | 2020-07-05 19:52:36,870 DEBUG: hub: exec hub --version [retcode=0]
weblate_1   | 2020-07-05 19:52:36,909 DEBUG: lab: exec lab --version [retcode=1]
weblate_1   | 2020-07-05 19:52:37,317 DEBUG: hg: exec hg version -q [retcode=0]
weblate_1   | 2020-07-05 19:52:38,270 DEBUG: Attempting to acquire lock 140188870381464 on /app/data/home/gitlock
weblate_1   | 2020-07-05 19:52:38,273 INFO: Lock 140188870381464 acquired on /app/data/home/gitlock
weblate_1   | 2020-07-05 19:52:38,287 DEBUG: git: exec git config --global merge.weblate-merge-gettext-po.name Weblate merge driver for Gettext PO files [retcode=0]
weblate_1   | 2020-07-05 19:52:38,305 DEBUG: git: exec git config --global merge.weblate-merge-gettext-po.driver /usr/local/lib/python3.7/dist-packages/weblate/examples/git-merge-gettext-po %O %A %B [retcode=0]
weblate_1   | 2020-07-05 19:52:38,320 DEBUG: git: exec git config --global user.email noreply@weblate.org [retcode=0]
weblate_1   | 2020-07-05 19:52:38,336 DEBUG: git: exec git config --global user.name Weblate [retcode=0]
weblate_1   | 2020-07-05 19:52:38,350 DEBUG: Using selector: EpollSelector
weblate_1   | 2020-07-05 19:52:38,371 DEBUG: Attempting to release lock 140188870381464 on /app/data/home/gitlock
weblate_1   | 2020-07-05 19:52:38,372 INFO: Lock 140188870381464 released on /app/data/home/gitlock

# NOW THE DOMAIN CHANGES

weblate_1   | 2020-07-05 19:52:39,958 INFO RPC interface 'supervisor' initialized
weblate_1   | 2020-07-05 19:52:39,959 INFO supervisord started with pid 1
weblate_1   | 2020-07-05 19:52:40,964 INFO spawned: 'stdout' with pid 313
weblate_1   | 2020-07-05 19:52:40,967 INFO spawned: 'celery-backup' with pid 314
weblate_1   | 2020-07-05 19:52:40,973 INFO spawned: 'celery-beat' with pid 315
weblate_1   | 2020-07-05 19:52:40,989 INFO spawned: 'celery-main' with pid 316
weblate_1   | 2020-07-05 19:52:40,996 INFO spawned: 'celery-memory' with pid 317
weblate_1   | 2020-07-05 19:52:41,010 INFO spawned: 'celery-notify' with pid 318
weblate_1   | 2020-07-05 19:52:41,029 INFO spawned: 'celery-translate' with pid 319
weblate_1   | 2020-07-05 19:52:41,048 INFO spawned: 'check' with pid 320
weblate_1   | 2020-07-05 19:52:41,084 INFO spawned: 'nginx' with pid 321
weblate_1   | 2020-07-05 19:52:41,112 INFO spawned: 'uwsgi' with pid 322
weblate_1   | 2020-07-05 19:52:42,742 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,743 INFO success: celery-backup entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,744 INFO success: celery-beat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,745 INFO success: celery-main entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,745 INFO success: celery-memory entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,746 INFO success: celery-notify entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,747 INFO success: celery-translate entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,747 INFO success: check entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,748 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
weblate_1   | 2020-07-05 19:52:42,748 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

This is about the point where changesite gets executed, but it should be changed to the domain matching WEBLATE_ALLOWED_HOSTS there. I've added verbose logging of this in https://github.com/WeblateOrg/docker/commit/57662b6a49de693df629acd1aa78cfb5ae0eab4b, so that it's clear what it does. It will be available as weblate/weblate:edge within few hours.

After switching to weblate/weblate:edge, to problem disappeared... And it is fixed in latest as well (there was an update after docker-compose pull). I'm sorry I can't reproduce it any more to figure out what caused it. I have made whole server snapshot and verified that the only change I did to fix it was the image update.

Thank you for your assistance.

The issue you have reported seems to be resolved now.

In case you see a similar problem, please open a separate issue.
If you are happy with the outcome, consider supporting Weblate by donating.

🤦 I just realised what was the problem. docker-compose restart doesn't reload settings in docker-compose.override.yml if the container already exists. Nor does docker-composer stop; docker-compose start. However, docker-compose stop; docker-compose up -d will re-create the container with the new values. So the the mysterious caching was simply an old set of environment variables tied to the container.

WeblateOrg / weblate