search

permitio / opal-helm-chart

You know, for Kubernetes
Apache License 2.0
21 stars 22 forks source link

OPAL Server breaks on startup if 'server.extraEnv.OPAL_SERVER_PORT' is missing #36

Closed maurice-freitag closed 1 year ago

maurice-freitag commented 1 year ago

I tried installing the chart (0.0.11) with helm install opal opal/opal after adding and updating the repo. The installed chart is version 0.0.11:

$ helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
opal    pva             1               2023-03-28 14:22:59.4348607 +0200 CEST  deployed        opal-0.0.11     0.5.0

The server never starts properly:

Logs: 

$ k logs -l  app.kubernetes.io/name=opal-server --tail 300 --follow
[2023-03-28 12:20:34 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-03-28 12:20:35 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
[2023-03-28 12:20:35 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2023-03-28 12:20:35 +0000] [7] [INFO] Booting worker with pid: 7
[2023-03-28 12:20:35 +0000] [8] [INFO] Booting worker with pid: 8
[2023-03-28 12:20:35 +0000] [9] [INFO] Booting worker with pid: 9
[2023-03-28 12:20:35 +0000] [10] [INFO] Booting worker with pid: 10
[2023-03-28 12:20:35 +0000] [7] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 8, in <module>
    app = create_app()
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 2, in create_app
    from .server import OpalServer
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/server.py", line 26, in <module>
    from opal_server.config import opal_server_config
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/config.py", line 288, in <module>
    opal_server_config = OpalServerConfig(prefix="OPAL_")
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 128, in __init__
    value = self._eval_and_save_entry(name, entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 162, in _eval_and_save_entry
    value = self._eval_entry(entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 168, in _eval_entry
    res = self._evaluate(whole_key, entry.default, entry.cast, **entry.kwargs)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 205, in _evaluate
    res = config(key, default=passed_default, cast=safe_cast_func, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 245, in __call__
    return self.config(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 107, in __call__
    return self.get(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 101, in get
    return cast(value)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 74, in wrapped_cast
    return cast_func(value, *args, **kwargs)
ValueError: invalid literal for int() with base 10: 'tcp://10.43.243.243:7002'
[2023-03-28 12:20:35 +0000] [7] [INFO] Worker exiting (pid: 7)
[2023-03-28 12:20:35 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 8, in <module>
    app = create_app()
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 2, in create_app
    from .server import OpalServer
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/server.py", line 26, in <module>
    from opal_server.config import opal_server_config
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/config.py", line 288, in <module>
    opal_server_config = OpalServerConfig(prefix="OPAL_")
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 128, in __init__
    value = self._eval_and_save_entry(name, entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 162, in _eval_and_save_entry
    value = self._eval_entry(entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 168, in _eval_entry
    res = self._evaluate(whole_key, entry.default, entry.cast, **entry.kwargs)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 205, in _evaluate
    res = config(key, default=passed_default, cast=safe_cast_func, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 245, in __call__
    return self.config(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 107, in __call__
    return self.get(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 101, in get
    return cast(value)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 74, in wrapped_cast
    return cast_func(value, *args, **kwargs)
ValueError: invalid literal for int() with base 10: 'tcp://10.43.243.243:7002'
[2023-03-28 12:20:35 +0000] [8] [INFO] Worker exiting (pid: 8)
[2023-03-28 12:20:35 +0000] [9] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 8, in <module>
    app = create_app()
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 2, in create_app
    from .server import OpalServer
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/server.py", line 26, in <module>
    from opal_server.config import opal_server_config
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/config.py", line 288, in <module>
    opal_server_config = OpalServerConfig(prefix="OPAL_")
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 128, in __init__
    value = self._eval_and_save_entry(name, entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 162, in _eval_and_save_entry
    value = self._eval_entry(entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 168, in _eval_entry
    res = self._evaluate(whole_key, entry.default, entry.cast, **entry.kwargs)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 205, in _evaluate
    res = config(key, default=passed_default, cast=safe_cast_func, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 245, in __call__
    return self.config(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 107, in __call__
    return self.get(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 101, in get
    return cast(value)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 74, in wrapped_cast
    return cast_func(value, *args, **kwargs)
ValueError: invalid literal for int() with base 10: 'tcp://10.43.243.243:7002'
[2023-03-28 12:20:35 +0000] [9] [INFO] Worker exiting (pid: 9)
[2023-03-28 12:20:35 +0000] [10] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.10/site-packages/uvicorn/workers.py", line 66, in init_process
    super(UvicornWorker, self).init_process()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 8, in <module>
    app = create_app()
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/main.py", line 2, in create_app
    from .server import OpalServer
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/server.py", line 26, in <module>
    from opal_server.config import opal_server_config
  File "/usr/local/lib/python3.10/site-packages/opal_server-0.5.0-py3.10.egg/opal_server/config.py", line 288, in <module>
    opal_server_config = OpalServerConfig(prefix="OPAL_")
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 128, in __init__
    value = self._eval_and_save_entry(name, entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 162, in _eval_and_save_entry
    value = self._eval_entry(entry)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 168, in _eval_entry
    res = self._evaluate(whole_key, entry.default, entry.cast, **entry.kwargs)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 205, in _evaluate
    res = config(key, default=passed_default, cast=safe_cast_func, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 245, in __call__
    return self.config(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 107, in __call__
    return self.get(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/decouple.py", line 101, in get
    return cast(value)
  File "/usr/local/lib/python3.10/site-packages/opal_common-0.5.0-py3.10.egg/opal_common/confi/confi.py", line 74, in wrapped_cast
    return cast_func(value, *args, **kwargs)
ValueError: invalid literal for int() with base 10: 'tcp://10.43.243.243:7002'
[2023-03-28 12:20:35 +0000] [10] [INFO] Worker exiting (pid: 10)
[2023-03-28 12:20:35 +0000] [1] [WARNING] Worker with pid 8 was terminated due to signal 15
[2023-03-28 12:20:35 +0000] [1] [WARNING] Worker with pid 10 was terminated due to signal 15
[2023-03-28 12:20:35 +0000] [1] [WARNING] Worker with pid 9 was terminated due to signal 15
[2023-03-28 12:20:35 +0000] [1] [INFO] Shutting down: Master
[2023-03-28 12:20:35 +0000] [1] [INFO] Reason: Worker failed to boot.

For some reason the application code attempts to parse 'tcp://10.43.243.243:7002' to an int and crashes understandibly. A workaround is to set server.extraEnv.OPAL_SERVER_PORT manually, either via helm install opal opal/opal --set server.extraEnv.OPAL_SERVER_PORT=7002 or a values.yaml file. The server then starts normally.

RazcoDev commented 1 year ago

Hey @maurice-freitag , IMO it's an issue with K8s is populating this env variable automatically, and OPAL uses it unintentionally.

Let me check this more deeply and i'll create a PR for this. Thanks for reporting this 👍

maurice-freitag commented 1 year ago

@RazcoDev FYI changing the name of the installed chart to myopal works as well (helm install myopal opal/opal) 🤠

RazcoDev commented 1 year ago

Makes sense, K8s is creating this env variable using the deployment/service name. In this case it's opal, and our "confi" settings using the environment variables with OPAL_ prefix.

Thanks !

roekatz commented 1 year ago

Fixed by https://github.com/permitio/opal/pull/442 You don't have to rename your chart :)