This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.
$ AIRFLOW_HOME=/var/lib/airflow airflow initdb [2019-09-07 20:51:32,416] {__init__.py:51} INFO - Using executor SequentialExecutor Traceback (most recent call last): File "/bin/airflow", line 22, in <module> from airflow.bin.cli import CLIFactory File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 68, in <module> from airflow.www_rbac.app import cached_app as cached_app_rbac File "/usr/lib/python2.7/site-packages/airflow/www_rbac/app.py", line 26, in <module> from flask_appbuilder import AppBuilder, SQLA File "/usr/lib/python2.7/site-packages/flask_appbuilder/__init__.py", line 5, in <module> from .base import AppBuilder File "/usr/lib/python2.7/site-packages/flask_appbuilder/base.py", line 5, in <module> from .api.manager import OpenApiManager File "/usr/lib/python2.7/site-packages/flask_appbuilder/api/__init__.py", line 11, in <module> from marshmallow_sqlalchemy.fields import Related, RelatedList File "/usr/lib/python2.7/site-packages/marshmallow_sqlalchemy/__init__.py", line 1, in <module> from .schema import TableSchemaOpts, ModelSchemaOpts, TableSchema, ModelSchema File "/usr/lib/python2.7/site-packages/marshmallow_sqlalchemy/schema.py", line 101 class TableSchema(ma.Schema, metaclass=TableSchemaMeta): ^ SyntaxError: invalid syntax
$ AIRFLOW_HOME=/var/lib/airflow airflow initdb [2019-09-07 20:51:32,416] {__init__.py:51} INFO - Using executor SequentialExecutor Traceback (most recent call last): File "/bin/airflow", line 22, in <module> from airflow.bin.cli import CLIFactory File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 68, in <module> from airflow.www_rbac.app import cached_app as cached_app_rbac File "/usr/lib/python2.7/site-packages/airflow/www_rbac/app.py", line 26, in <module> from flask_appbuilder import AppBuilder, SQLA File "/usr/lib/python2.7/site-packages/flask_appbuilder/__init__.py", line 5, in <module> from .base import AppBuilder File "/usr/lib/python2.7/site-packages/flask_appbuilder/base.py", line 5, in <module> from .api.manager import OpenApiManager File "/usr/lib/python2.7/site-packages/flask_appbuilder/api/__init__.py", line 11, in <module> from marshmallow_sqlalchemy.fields import Related, RelatedList File "/usr/lib/python2.7/site-packages/marshmallow_sqlalchemy/__init__.py", line 1, in <module> from .schema import TableSchemaOpts, ModelSchemaOpts, TableSchema, ModelSchema File "/usr/lib/python2.7/site-packages/marshmallow_sqlalchemy/schema.py", line 101 class TableSchema(ma.Schema, metaclass=TableSchemaMeta): ^ SyntaxError: invalid syntaxDowngrading marshmallow version also doesn't help