helm / charts

⚠️(OBSOLETE) Curated applications for Kubernetes
Apache License 2.0
15.49k stars 16.8k forks source link

[stable/airflow] Flower gives postgres error logs after starting up #15869

Closed deepaksood619 closed 5 years ago

deepaksood619 commented 5 years ago

Describe the bug A clear and concise description of what the bug is. When i create a new airflow deployment this error comes in flower logs

Version of Helm and Kubernetes: helm version

Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

kubectl version

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Which chart: stable/airflow

What happened: Error while deploying airflow with default values

What you expected to happen: no errors

How to reproduce it (as minimally and precisely as possible): helm install --name air -f airflow/values.yaml --namespace airflow stable/airflow --version 3.0.2 kubectl logs -f air-flower-55cb567664-hxhm6 -n airflow

Anything else we need to know:

Thu Jul 25 07:02:54 UTC 2019 - waiting for Postgres... 1/20
[2019-07-25 07:03:10,918] {{settings.py:182}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=1
[2019-07-25 07:03:11,294] {{__init__.py:51}} INFO - Using executor CeleryExecutor
[2019-07-25 07:03:11,677] {{cli_action_loggers.py:70}} ERROR - Failed on pre-execution callback using <function default_action_log at 0x7f89fb4a1e18>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "log" does not exist
LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
                    ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 68, in on_pre_execution
    cb(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/cli_action_loggers.py", line 99, in default_action_log
    session.add(log)
  File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 44, in create_session
    session.commit()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1023, in commit
    self.transaction.commit()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 487, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 466, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2446, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2584, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 67, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 277, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2544, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 416, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 583, in execute
    uow,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    insert,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements
    statement, params
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 980, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
    distilled_params,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "log" does not exist
LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
                    ^
 [SQL: 'INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s, %(task_id)s, %(event)s, %(execution_date)s, %(owner)s, %(extra)s) RETURNING log.id'] [parameters: {'dttm': datetime.datetime(2019, 7, 25, 7, 3, 11, 537214, tzinfo=<Timezone [UTC]>), 'dag_id': None, 'task_id': None, 'event': 'cli_flower', 'execution_date': None, 'owner': 'airflow', 'extra': '{"host_name": "air-flower-55cb567664-hxhm6", "full_command": "[\'/usr/local/bin/airflow\', \'flower\']"}'}] (Background on this error at: http://sqlalche.me/e/f405)
[I 190725 07:03:12 command:136] Visit me at http://0.0.0.0:5555
[I 190725 07:03:12 command:141] Broker: redis://:**@air-redis-master:6379/1
[I 190725 07:03:12 command:144] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']
[I 190725 07:03:12 mixins:229] Connected to redis://:**@air-redis-master:6379/1
[W 190725 07:03:17 control:44] 'stats' inspect method failed
[W 190725 07:03:17 control:44] 'active_queues' inspect method failed
[W 190725 07:03:17 control:44] 'registered' inspect method failed
[W 190725 07:03:17 control:44] 'scheduled' inspect method failed
[W 190725 07:03:17 control:44] 'active' inspect method failed
[W 190725 07:03:17 control:44] 'reserved' inspect method failed
[W 190725 07:03:17 control:44] 'revoked' inspect method failed
[W 190725 07:03:17 control:44] 'conf' inspect method failed
dragid10 commented 5 years ago

Did you ever figure out what was wrong, I'm getting this exact same issue

dragid10 commented 5 years ago

Actually running kubectl exec -it <POD> airflow initdb --namespace=<YOUR-NAMESPACE> helped me fix that issue. I don't know why the db isn't being initialized the first time properly though

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

stale[bot] commented 5 years ago

This issue is being automatically closed due to inactivity.