dockerapi errors on start up after upgrade to 2.x, works rolling back to 1.x

JFossey commented 1 year ago

Contribution guidelines

[X] I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
[X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
[X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
[X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

After seeing your recent security advisory we decided to upgrade our company mailcow install. Last time we upgraded was Dec 2022.

After the upgrade nothing seemed to work. PHP-FPM was not starting fully and just hanging, rspamd logged it was waiting for PHP. After checking the logs of different services we found the dockerapi service was showing an error.

Restarting individual services did not work, restarting the full stack did not work. Eventually we rolled back just the dockerapi service from `2.02` to `1.43` and restarted the full stack and everything now seems to work.

I am sure there will be backward compatibility issues and some API calls will not work, but most things seem to be working and we are back up and running for now.

Logs:

dockerapi-mailcow_1   | -----
dockerapi-mailcow_1   | Traceback (most recent call last):
dockerapi-mailcow_1   |   File "/usr/bin/uvicorn", line 8, in <module>
dockerapi-mailcow_1   |     sys.exit(main())
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
dockerapi-mailcow_1   |     return self.main(*args, **kwargs)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1055, in main
dockerapi-mailcow_1   |     rv = self.invoke(ctx)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
dockerapi-mailcow_1   |     return ctx.invoke(self.callback, **ctx.params)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/click/core.py", line 760, in invoke
dockerapi-mailcow_1   |     return __callback(*args, **kwargs)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/uvicorn/main.py", line 403, in main
dockerapi-mailcow_1   |     run(
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/uvicorn/main.py", line 568, in run
dockerapi-mailcow_1   |     server.run()
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/uvicorn/server.py", line 59, in run
dockerapi-mailcow_1   |     return asyncio.run(self.serve(sockets=sockets))
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
dockerapi-mailcow_1   |     return loop.run_until_complete(main)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
dockerapi-mailcow_1   |     return future.result()
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/uvicorn/server.py", line 66, in serve
dockerapi-mailcow_1   |     config.load()
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/uvicorn/config.py", line 471, in load
dockerapi-mailcow_1   |     self.loaded_app = import_from_string(self.app)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
dockerapi-mailcow_1   |     module = importlib.import_module(module_str)
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
dockerapi-mailcow_1   |     return _bootstrap._gcd_import(name[level:], package, level)
dockerapi-mailcow_1   |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
dockerapi-mailcow_1   |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
dockerapi-mailcow_1   |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
dockerapi-mailcow_1   |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
dockerapi-mailcow_1   |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
dockerapi-mailcow_1   |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
dockerapi-mailcow_1   |   File "/app/dockerapi.py", line 536, in <module>
dockerapi-mailcow_1   |     if os.environ['REDIS_SLAVEOF_IP'] != "":
dockerapi-mailcow_1   |   File "/usr/lib/python3.10/os.py", line 680, in __getitem__
dockerapi-mailcow_1   |     raise KeyError(key) from None
dockerapi-mailcow_1   | KeyError: 'REDIS_SLAVEOF_IP'

Steps to reproduce:

1. Have a working mailcow install that is more than 4 months old.
2. Upgrade to the latest master as of the 5th of April.
3. No ENV changes were made.
4. After stopping and starting all services check `dockerapi-mailcow` logs.

Which branch are you using?

master

Operating System:

Debian

Server/VM specifications:

4G, 1 CPU

Is Apparmor, SELinux or similar active?

yes

Virtualization technology:

KVM

Docker version:

20.10.22

docker-compose version or docker compose version:

1.25.0

mailcow version:

2023-04a

Reverse proxy:

HAProxy

Logs of git diff:

N/A

Logs of iptables -L -vn:

N/A

Logs of ip6tables -L -vn:

N/A

Logs of iptables -L -vn -t nat:

N/A

Logs of ip6tables -L -vn -t nat:

N/A

DNS check:

N/A

MAGICCC commented 1 year ago

Did you somehow remove REDIS_SLAVEOF_IP enviroment var from your compose file (judging from the error message stating the env variable cannot be found/not defined)

JFossey commented 1 year ago

@MAGICCC :100:, that was it.

Because of some customisation's we manually manage our compose file and it seems we missed the 2 REDIS envs being added last year, but them being missing was only triggering an error in the 2.x version of the docker api container.

Thank you for your time and prompt response.

JFossey commented 1 year ago

I doubt anyone else would make the same mistake, but if anyone does this is what we missed.

Related Symptoms:

Sogo-mailcow. Waiting for schema update
Rspamd-mailcow. Waiting for php on port 9001

Related missing Docker Api container compose config:

mailcow / mailcow-dockerized