Can't run zulip from docker container / Fatal Python error

luisgois commented 5 years ago

Hi.

I'm launching each container individually (I don't have docker-compose available):

docker run -d --name database --net=test -e POSTGRES_DB=zulip -e POSTGRES_USER=zulip -e POSTGRES_PASSWORD=zulip -v /opt/docker/zulip/postgresql/new:/var/lib/postgresql/data:rw zulip/zulip-postgresql:latest
docker run -d --name memcached --net=test sameersbn/memcached
docker run -d --name rabitmq --net=test -h zulip-rabbit -e RABBITMQ_DEFAULT_USER=zulip -e RABBITMQ_DEFAULT_PASS=zulip -v /opt/docker/zulip/rabbitmq:/var/lib/rabbitmq:rw library/rabbitmq:3.7.7
docker run -d --name redis --net=test -v /opt/docker/zulip/redis:/var/lib/redis:rw sameersbn/redis
docker run -d --name zulip --ulimit nofile=40000 --net=test -e SSL_CERTIFICATE_GENERATION -e SETTING_EXTERNAL_HOST -e SETTING_ZULIP_ADMINISTRATOR -e DEBUG -e DB_HOST=database -e POSTGRES_PASSWORD=zulip -e SECRETS_postgres_password=zulip zulip/docker-zulip

When zulip is initializing the database I get a "Fatal Python error" as shown below:

 (...)
 Applying zerver.0189_userprofile_add_some_emojisets... OK
  Applying zerver.0190_cleanup_pushdevicetoken... OK
+ ./manage.py createcachetable third_party_api_results
+ ./manage.py initialize_voyager_db
Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007ffff7fef700 (most recent call first):
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 463 in __init__
  File "/usr/lib/python3.5/traceback.py", line 117 in format_exception
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/logging_handlers.py", line 114 in emit
  File "/usr/lib/python3.5/logging/__init__.py", line 855 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  File "/usr/lib/python3.5/logging/__init__.py", line 1308 in error
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django_pylibmc/memcached.py", line 146 in set
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/lib/logging_util.py", line 31 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 710 in filter
  File "/usr/lib/python3.5/logging/__init__.py", line 851 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1487 in callHandlers
  File "/usr/lib/python3.5/logging/__init__.py", line 1425 in handle
  File "/usr/lib/python3.5/logging/__init__.py", line 1415 in _log
  ...
/home/zulip/deployments/current/scripts/setup/initialize-database: line 44:    79 Aborted                 ./manage.py initialize_voyager_db
+ set +x

Populating default database failed.
After you fix the problem, you will need to do the following before rerunning this:
  * supervisorctl stop all # to stop all services that might be accessing the database
  * scripts/setup/postgres-init-db # run as root to drop and re-create the database

Zulip first start database initi failed in "initialize-database" exit code 1. Exiting.
+ RETURN_CODE=1
+ [[ 1 != 0 ]]
+ echo 'Zulip first start database initi failed in "initialize-database" exit code 1. Exiting.'
+ exit 1

timabbott commented 5 years ago

@luisgois I've never seen this failure mode before. "Fatal Python error: Cannot recover from stack overflow." in the Python traceback module suggests some sort of exception is being thrown that throws more exceptions. Line 31 of logging_util.py is part of Zulip's exception reporting codebase that is supposed to talk to memcached, so my guess is at least part of your problem is that your memcached host is not up at the time this code is running (or not reachable from the zulip host).

Can you talk about why you can't install docker-compose in your environment?

luisgois commented 5 years ago

Not having docker-compose is a decision beyond my control. I cannot install it or ask for it without a long wait.

Nevertheless, your comment was helpful. I've now started zulip with "-e SETTING_MEMCACHED_LOCATION=memcached:11211" and it passed past the point where it was failing. I assumed the entrypoint.sh was assuming the default values of the docker-compose.yml but it isn't. This is confusing and made me spend lots of time (docker newbie here) to figure out what was wrong and the command line options required.

Thanks anyway. Cheers,

Luis Gois

luisgois commented 5 years ago

Eventually got over all errors by setting all environment variables mentioned in docker-compose.yml. however, still 2 problems, already after creating the organization:

can't send mails with SMTP. JIRA on the same docker host works fine with smtp host "localhost" and port "25" (no user or password required).
can't add a new bot (incoming webhook to try integration with JIRA). always get "bad name or username" no matter what I choose for those values.

How can I turn on additional logging? Couldn't find any error message.

galexrt commented 5 years ago

@luisgois

Every container has its own localhost. You need to specify the full SMTP server address to be used in containers (there are other ways but they are "messy" in point of security).

luisgois commented 5 years ago

Indeed, sorry for the misunderstanding. In the settings.py, the EMAIL_HOST contains the fully qualified hostname of the docker host.

I've used ./manage.py send_test_email email@mydomain and got the errror:

Traceback (most recent call last):
  File "./manage.py", line 46, in <module>
    execute_from_command_line(sys.argv)
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/zulip/deployments/2018-11-20-04-17-50/zerver/management/commands/send_test_email.py", line 21, in handle
    send_mail("Zulip email test", message, FromAddress.SUPPORT, kwargs['email'])
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/mail/__init__.py", line 62, in send_mail
    return mail.send()
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/mail/message.py", line 348, in send
    return self.get_connection(fail_silently).send_messages([self])
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/mail/backends/smtp.py", line 104, in send_messages
    new_conn_created = self.open()
  File "/home/zulip/deployments/2018-11-20-04-17-50/zulip-py3-venv/lib/python3.5/site-packages/django/core/mail/backends/smtp.py", line 64, in open
    self.connection = self.connection_class(self.host, self.port, **connection_params)
  File "/usr/lib/python3.5/smtplib.py", line 251, in __init__
    (code, msg) = self.connect(host, port)
  File "/usr/lib/python3.5/smtplib.py", line 335, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib/python3.5/smtplib.py", line 306, in _get_socket
    self.source_address)
  File "/usr/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/usr/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

luisgois commented 5 years ago

As an update, changed the email backend to the django console, asked for a password reset and now even the bot creation works. I'm just trying zulip out in a test environment. Do you know what might be going on? Next step is integration with JIRA and Jenkins.

In production we'll need ldap authentication and smtp. Maybe we won't use docker and go for installation on our servers, the problem is that we only use RedHat Enterprise, not Ubuntu.

timabbott commented 1 day ago

I don't think there'a way to figure out what went wrong here, but it seems weird enough to not be worth further attention.

zulip / docker-zulip

Can't run zulip from docker container / Fatal Python error #169

Thanks anyway. Cheers,