search

oomichi / try-kubernetes

12 stars 5 forks source link

Deploy Apache Airflow #106

Closed oomichi closed 4 years ago

oomichi commented 4 years ago

ubuntu 18.04 VM状に以下の手順で構築する。

https://airflow.apache.org/start.html

# install from pypi using pip
pip install apache-airflow

# initialize the database
airflow initdb

# start the web server, default port is 8080
airflow webserver -p 8080

# start the scheduler
airflow scheduler
oomichi commented 4 years ago

initdb で以下のエラーが発生。

$ sudo airflow initdb
[2019-10-14 21:25:33,850] {__init__.py:51} INFO - Using executor SequentialExecutor
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 22, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 68, in <module>
    from airflow.www_rbac.app import cached_app as cached_app_rbac
  File "/usr/local/lib/python2.7/dist-packages/airflow/www_rbac/app.py", line 26, in <module>
    from flask_appbuilder import AppBuilder, SQLA
  File "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/__init__.py", line 5, in <module>
    from .base import AppBuilder
  File "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/base.py", line 5, in <module>
    from .api.manager import OpenApiManager
  File "/usr/local/lib/python2.7/dist-packages/flask_appbuilder/api/__init__.py", line 11, in <module>
    from marshmallow_sqlalchemy.fields import Related, RelatedList
  File "/usr/local/lib/python2.7/dist-packages/marshmallow_sqlalchemy/__init__.py", line 1, in <module>
    from .schema import TableSchemaOpts, ModelSchemaOpts, TableSchema, ModelSchema
  File "/usr/local/lib/python2.7/dist-packages/marshmallow_sqlalchemy/schema.py", line 101
    class TableSchema(ma.Schema, metaclass=TableSchemaMeta):
                                          ^
SyntaxError: invalid syntax

どう見ても一般的な実装エラーに見えるので Issue をトラッキングする。 Airflow は github 状で管理されている。 https://github.com/apache/airflow

Stackflow で同問題がレポートされていた。 どうやらmarshmallow-sqlalchemyのバージョンがダメらしい（非互換変更発生？）

pip uninstall marshmallow-sqlalchemy
pip install marshmallow-sqlalchemy==0.17.1

上記 Stackoverflow と同様に 0.19.0 がインストールされている。

$ sudo pip show marshmallow-sqlalchemy
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Name: marshmallow-sqlalchemy
Version: 0.19.0
Summary: SQLAlchemy integration with the marshmallow (de)serialization library
Home-page: https://github.com/marshmallow-code/marshmallow-sqlalchemy
Author: Steven Loria
Author-email: sloria1@gmail.com
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires: SQLAlchemy, marshmallow
Required-by: Flask-AppBuilder

上記の手順で問題解消

oomichi commented 4 years ago

サービスが立ちあがるところまで成功した。