Playbook is failing on check admin user(>2.0)

[rajaniesh]

Description

[Description of the issue]

Steps to Reproduce

Just run the playbook for ubuntu 18.04 [What you expect to happen] I was expecting it to deploy Airflow without any error Actual behavior: TASK [deploy_airflow : Airflow | Check Admin user (> 2.0)] ***** fatal: [dmpServer]: FAILED! => {"changed": false, "cmd": ["/opt/airflow/bin/airflow", "users", "list"], "delta": "0:00:04.620829", "end": "2021-06-22 01:24:22.929163", "msg": "non-zero return code", "rc": 1, "start": "2021-06-22 01:24:18.308334", "stderr": "/opt/airflow/lib/python3.6/site-packages/flask_appbuilder/models/sqla/interface.py:62 SAWarning: relationship 'DagRun.serialized_dag' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id), 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. To silence this warning, add the parameter 'overlaps=\"dag_run,task_instances\"' to the 'DagRun.serialized_dag' relationship.\n/opt/airflow/lib/python3.6/site-packages/flask_appbuilder/models/sqla/interface.py:62 SAWarning: relationship 'SerializedDagModel.dag_runs' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id), 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. To silence this warning, add the parameter 'overlaps=\"dag_run,task_instances\"' to the 'SerializedDagModel.dag_runs' relationship.\nTraceback (most recent call last):\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 209, in add_paths\n self.add_operation(path, method)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 173, in add_operation\n pass_context_arg_name=self.pass_context_arg_name\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/init.py\", line 8, in make_operation\n return spec.operation_cls.from_spec(spec, *args, kwargs)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/openapi.py\", line 138, in from_spec\n kwargs\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/openapi.py\", line 89, in init\n pass_context_arg_name=pass_context_arg_name\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/abstract.py\", line 96, in init\n self._resolution = resolver.resolve(self)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/resolver.py\", line 40, in resolve\n return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/resolver.py\", line 66, in resolve_function_from_operation_id\n raise ResolverError(str(e), sys.exc_info())\nconnexion.exceptions.ResolverError: \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/airflow/bin/airflow\", line 8, in \n sys.exit(main())\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/main.py\", line 40, in main\n args.func(args)\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/cli/cli_parser.py\", line 48, in command\n return func(*args, *kwargs)\n File \"/usr/lib/python3.6/contextlib.py\", line 52, in inner\n return func(args, kwds)\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/cli/commands/user_command.py\", line 35, in users_list\n appbuilder = cached_app().appbuilder # pylint: disable=no-member\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/www/app.py\", line 135, in cached_app\n app = create_app(config=config, testing=testing)\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/www/app.py\", line 120, in create_app\n init_api_connexion(flask_app)\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/www/extensions/init_views.py\", line 172, in init_api_connexion\n specification='v1.yaml', base_path=base_path, validate_responses=True, strict_validation=True\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apps/flask_app.py\", line 57, in add_api\n api = super(FlaskApp, self).add_api(specification, kwargs)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apps/abstract.py\", line 156, in add_api\n options=api_options.as_dict())\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 111, in init\n self.add_paths()\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 216, in add_paths\n self._handle_add_operation_error(path, method, err.exc_info)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 231, in _handle_add_operation_error\n raise value.with_traceback(traceback)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/resolver.py\", line 61, in resolve_function_from_operation_id\n return self.function_resolver(operation_id)\n File \"/opt/airflow/lib/python3.6/site-packages/connexion/utils.py\", line 111, in get_function_from_name\n module = importlib.import_module(module_name)\n File \"/usr/lib/python3.6/importlib/init.py\", line 126, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File \"\", line 994, in _gcd_import\n File \"\", line 971, in _find_and_load\n File \"\", line 955, in _find_and_load_unlocked\n File \"\", line 665, in _load_unlocked\n File \"\", line 678, in exec_module\n File \"\", line 219, in _call_with_frames_removed\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/api_connexion/endpoints/connection_endpoint.py\", line 26, in \n from airflow.api_connexion.schemas.connection_schema import (\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/api_connexion/schemas/connection_schema.py\", line 42, in \n class ConnectionSchema(ConnectionCollectionItemSchema): # pylint: disable=too-many-ancestors\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow/schema.py\", line 117, in new\n dict_cls=dict_cls,\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 94, in get_declared_fields\n fields.update(mcs.get_auto_fields(fields, converter, opts, dict_cls))\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 108, in get_auto_fields\n for field_name, field in fields.items()\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 110, in \n and field_name not in opts.exclude\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 28, in create_field\n return converter.field_for(model, column_name, self.field_kwargs)\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/convert.py\", line 171, in field_for\n return self.property2field(prop, kwargs)\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/convert.py\", line 146, in property2field\n field_class = field_class or self._get_field_class_for_property(prop)\n File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/convert.py\", line 210, in _get_field_class_for_property\n column = prop.columns[0]\n File \"/opt/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py\", line 1240, in getattr\n return self._fallback_getattr(key)\n File \"/opt/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py\", line 1214, in _fallback_getattr\n raise AttributeError(key)\nAttributeError: columns", "stderr_lines": ["/opt/airflow/lib/python3.6/site-packages/flask_appbuilder/models/sqla/interface.py:62 SAWarning: relationship 'DagRun.serialized_dag' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id), 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. To silence this warning, add the parameter 'overlaps=\"dag_run,task_instances\"' to the 'DagRun.serialized_dag' relationship.", "/opt/airflow/lib/python3.6/site-packages/flask_appbuilder/models/sqla/interface.py:62 SAWarning: relationship 'SerializedDagModel.dag_runs' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id), 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. To silence this warning, add the parameter 'overlaps=\"dag_run,task_instances\"' to the 'SerializedDagModel.dag_runs' relationship.", "Traceback (most recent call last):", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 209, in add_paths", " self.add_operation(path, method)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 173, in add_operation", " pass_context_arg_name=self.pass_context_arg_name", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/init.py\", line 8, in make_operation", " return spec.operation_cls.from_spec(spec, args, kwargs)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/openapi.py\", line 138, in from_spec", " kwargs", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/openapi.py\", line 89, in init", " pass_context_arg_name=pass_context_arg_name", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/operations/abstract.py\", line 96, in init", " self._resolution = resolver.resolve(self)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/resolver.py\", line 40, in resolve", " return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/resolver.py\", line 66, in resolve_function_from_operation_id", " raise ResolverError(str(e), sys.exc_info())", "connexion.exceptions.ResolverError: ", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/opt/airflow/bin/airflow\", line 8, in ", " sys.exit(main())", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/main.py\", line 40, in main", " args.func(args)", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/cli/cli_parser.py\", line 48, in command", " return func(args, kwargs)", " File \"/usr/lib/python3.6/contextlib.py\", line 52, in inner", " return func(*args, kwds)", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/cli/commands/user_command.py\", line 35, in users_list", " appbuilder = cached_app().appbuilder # pylint: disable=no-member", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/www/app.py\", line 135, in cached_app", " app = create_app(config=config, testing=testing)", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/www/app.py\", line 120, in create_app", " init_api_connexion(flask_app)", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/www/extensions/init_views.py\", line 172, in init_api_connexion", " specification='v1.yaml', base_path=base_path, validate_responses=True, strict_validation=True", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apps/flask_app.py\", line 57, in add_api", " api = super(FlaskApp, self).add_api(specification, kwargs)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apps/abstract.py\", line 156, in add_api", " options=api_options.as_dict())", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 111, in init", " self.add_paths()", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 216, in add_paths", " self._handle_add_operation_error(path, method, err.exc_info)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/apis/abstract.py\", line 231, in _handle_add_operation_error", " raise value.with_traceback(traceback)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/resolver.py\", line 61, in resolve_function_from_operation_id", " return self.function_resolver(operation_id)", " File \"/opt/airflow/lib/python3.6/site-packages/connexion/utils.py\", line 111, in get_function_from_name", " module = importlib.import_module(module_name)", " File \"/usr/lib/python3.6/importlib/init.py\", line 126, in import_module", " return _bootstrap._gcd_import(name[level:], package, level)", " File \"\", line 994, in _gcd_import", " File \"\", line 971, in _find_and_load", " File \"\", line 955, in _find_and_load_unlocked", " File \"\", line 665, in _load_unlocked", " File \"\", line 678, in exec_module", " File \"\", line 219, in _call_with_frames_removed", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/api_connexion/endpoints/connection_endpoint.py\", line 26, in ", " from airflow.api_connexion.schemas.connection_schema import (", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/api_connexion/schemas/connection_schema.py\", line 42, in ", " class ConnectionSchema(ConnectionCollectionItemSchema): # pylint: disable=too-many-ancestors", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow/schema.py\", line 117, in new", " dict_cls=dict_cls,", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 94, in get_declared_fields", " fields.update(mcs.get_auto_fields(fields, converter, opts, dict_cls))", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 108, in get_auto_fields", " for field_name, field in fields.items()", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 110, in ", " and field_name not in opts.exclude", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py\", line 28, in create_field", " return converter.field_for(model, column_name, self.field_kwargs)", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/convert.py\", line 171, in field_for", " return self.property2field(prop, kwargs)", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/convert.py\", line 146, in property2field", " field_class = field_class or self._get_field_class_for_property(prop)", " File \"/opt/airflow/lib/python3.6/site-packages/marshmallow_sqlalchemy/convert.py\", line 210, in _get_field_class_for_property", " column = prop.columns[0]", " File \"/opt/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py\", line 1240, in getattr", " return self._fallback_getattr(key)", " File \"/opt/airflow/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py\", line 1214, in _fallback_getattr", " raise AttributeError(key)", "AttributeError: columns"], "stdout": "", "stdout_lines": []}

Reproduces how often: [What percentage of the time does it reproduce?] 100%

Environment

Ubuntu 18.04

• The release version/s you are using:
• OS:
• Others:

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue.]

comments

In my machine I already have the MySQL and Tomcat installed and now I want to install Airflow onto it.

[ultraheroe]

Please, can you fill up our bug issue template?

[rajaniesh]

Sorry I forgot to remove the comments from bug template. It is updated now

[rajaniesh]

Sir, any clue!!

[ultraheroe]

@rajaniesh I'm trying to reproduce the bug, we usually run this role in debian10 machines, please be patient while I'm testing it on a ubuntu 18.04 😉

[rajaniesh]

Thanks a lot for your help I will wait for your feedback

[ultraheroe]

Hello @rajaniesh I was running a local test with molecule and ubuntu:18.04 docker image. The result was a run without errors and running well.

Now I'll try to install with mysql extra package and see what happens

So, for further support, can you give us more information about how are you using the role in your playbook? Thank you

[rajaniesh]

Sir,

I am adding this as a role in my playbook and running it. I have mysql and tomcat already installed and I want to use them as it is. I want to use existing mysql db used as airflow db and tomcat to host the ui with 8085 port because I am using 8080. Regards Rajaniesh

[rajaniesh]

How long it may take to conduct test?

[rajaniesh]

Hi,

I again ran it today and it threw this error:

TASK [airflow-role-master : Airflow | Initializing DB > 2.0] ***** fatal: [dmpServer]: FAILED! => {"changed": true, "cmd": ["/opt/airflow/bin/airflow", "db", "init"], "delta": "0:00:01.832997", "end": "2021-06-25 15:27:39.436739", "msg": "non-zero return code", "rc": 1, "start": "2021-06-25 15:27:37.603742", "stderr": "Traceback (most recent call last):\n File \"/usr/lib/python3.6/pathlib.py\", line 1248, in mkdir\n self._accessor.mkdir(self, mode)\n File \"/usr/lib/python3.6/pathlib.py\", line 387, in wrapped\n return strfunc(str(pathobj), args)\nFileNotFoundError: [Errno 2] No such file or directory: '/home/airflow/airflow'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/opt/airflow/bin/airflow\", line 5, in \n from airflow.main import main\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/init.py\", line 34, in \n from airflow import settings\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/settings.py\", line 37, in \n from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401\n File \"/opt/airflow/lib/python3.6/site-packages/airflow/configuration.py\", line 736, in \n pathlib.Path(AIRFLOW_HOME).mkdir(parents=True, exist_ok=True)\n File \"/usr/lib/python3.6/pathlib.py\", line 1252, in mkdir\n self.parent.mkdir(parents=True, exist_ok=True)\n File \"/usr/lib/python3.6/pathlib.py\", line 1248, in mkdir\n self._accessor.mkdir(self, mode)\n File \"/usr/lib/python3.6/pathlib.py\", line 387, in wrapped\n return strfunc(str(pathobj), args)\nPermissionError: [Errno 13] Permission denied: '/home/airflow'", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib/python3.6/pathlib.py\", line 1248, in mkdir", " self._accessor.mkdir(self, mode)", " File \"/usr/lib/python3.6/pathlib.py\", line 387, in wrapped", " return strfunc(str(pathobj), args)", "FileNotFoundError: [Errno 2] No such file or directory: '/home/airflow/airflow'", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/opt/airflow/bin/airflow\", line 5, in ", " from airflow.main import main", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/init.py\", line 34, in ", " from airflow import settings", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/settings.py\", line 37, in ", " from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf # NOQA F401", " File \"/opt/airflow/lib/python3.6/site-packages/airflow/configuration.py\", line 736, in ", " pathlib.Path(AIRFLOW_HOME).mkdir(parents=True, exist_ok=True)", " File \"/usr/lib/python3.6/pathlib.py\", line 1252, in mkdir", " self.parent.mkdir(parents=True, exist_ok=True)", " File \"/usr/lib/python3.6/pathlib.py\", line 1248, in mkdir", " self._accessor.mkdir(self, mode)", " File \"/usr/lib/python3.6/pathlib.py\", line 387, in wrapped", " return strfunc(str(pathobj), args)", "PermissionError: [Errno 13] Permission denied: '/home/airflow'"], "stdout": "", "stdout_lines": []}

Regards Rajaniesh

[ultraheroe]

Remember to read de README.md

Try to setup your AIRFLOW_HOME and AIRFLOW_CONFIG (looks like you don't have write permission) Look at the three vars files of the role described here https://github.com/idealista/airflow-role#usage-runner to correctly set up your playbook, for example to use other webserver port you definitely must set the airflow_webserver_port var

[RyanQuey]

I am getting this error too, even after setting AIRFLOW_HOME and AIRFLOW_CONFIG.

What fixes it for me at least is to change the tasks in tasks/users.yml to use

become: true
become_user: "{{ airflow_user }}"

Doing this for both Airflow | Check Admin user (> 2.0) and Airflow | Create Admin user AUTH_DB (> 2.0) makes it work.

[ultraheroe]

Hello @RyanQuey, can I ask what Os and version are you using? ubuntu like Rajaniesh too?

[RyanQuey]

Hello @ultraheroe yes, Ubuntu 18 for both Ansible controller and target host.

[Sureya]

Hi @ultraheroe ,

I dont yet see 2.1.1 in ansible galaxy page. How do i go about installing the latest changes please ? I am also facing the same issue

[vicsufer]

Hi @Sureya, It seems that yesterday travis had some issues with triggering builds. We're going throught the release pipeline again to publish the new version. An aditional note, the release tag will be 2.0.3 not 2.1.1.

[Sureya]

@vicsufer : Many thanks for your response. Looking forward for the release 👍🏾

[vicsufer]

@Sureya Release 2.0.3 is already available on ansible-galaxy, thanks for your patience.

[Sureya]

@vicsufer : Many thanks for the reply. I can confirm that my airflow works fine on Ubuntu.