search

villasv / aws-airflow-stack

Turbine: the bare metals that gets you Airflow
https://victor.villas/aws-airflow-stack/
MIT License
377 stars 69 forks source link

Setup fails on eu-central-1 due to AccessDenied when calling the ListObjectsV2 #180

Closed ilyanoskov closed 4 years ago

ilyanoskov commented 4 years ago

After running your CloudFormation template I am unable to access the webserver running on 8080 port. It just does not connect. I tried running the template two times to no avail

ilyanoskov commented 4 years ago

@villasv I am also unable to SSH into webserver even with EC2 Instance Connect

villasv commented 4 years ago

Hi, @ilyanoskov. I don't remember testing with eu-central-1, but I'm going to check it.

So it looks like the CloudFormation deploymed finished successfully, then? And the Webserver not responding is what you're reporting?

As per SSH, SSH is not enabled by default. You would have to use Session Manager.

villasv commented 4 years ago

I could not reproduce. I successfully deployed the stack on eu-central-1 and was able to access the webserver UI. Can you post here the creation parameters?

Also, could you tell me if youre deploying the master template or the cluster template?

ilyanoskov commented 4 years ago

@villasv thanks a lot for getting back to me, I am desperate at this point haha. I am deploying it now, here are the parameters:

Key | Value
-- | --
AllowHTTPAccessCIDR | 10.0.0.0/16
GrowthThreshold | 0.9
LoadDefaultCons | False
LoadExampleDags | True
MaxGroupSize | 10
MinGroupSize | 0
PrivateSubnet1ACIDR | 10.0.10.0/24
PrivateSubnet2ACIDR | 10.0.11.0/24
PublicSubnet1CIDR | 10.0.0.0/24
PublicSubnet2CIDR | 10.0.1.0/24
QSS3BucketName | turbine-quickstart
QSS3KeyPrefix | quickstart-turbine-airflow/
SchedulerInstanceType | t3.micro
ShrinkThreshold | 0.5
VPCCIDR | 10.0.0.0/16
WebServerPort | 8080
WebserverInstanceType | t3.micro
WorkerInstanceType | t3.medium
ilyanoskov commented 4 years ago

@villasv yes, I have recreated it again and I am unable to access the webserver UI. All I did was press CloudFormation button (deploying the master template), wait for it to finish (all steps finished successfully), wait 3-5 mins for airflow to startup on ec2, and then try to access ec2-x-x-x-x-x.eu-central-1.compute.amazonaws.com:8080 to no avail. What is the best way to debug this?

The system log of webserver showed this:

[   18.876158] cloud-init[2647]: + /opt/aws/bin/cfn-init -v --region eu-central-1 --stack airflow-stack-TurbineCluster-NUXXXXXXX-WebserverStack-DPSXXXXXX --resource LaunchConfiguration
[   20.432501] cloud-init[2647]: Error occurred during build: Command setup failed
[   20.461524] cloud-init[2647]: May 17 19:23:47 cloud-init[2647]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
[   20.465338] cloud-init[2647]: May 17 19:23:47 cloud-init[2647]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
[   20.467739] cloud-init[2647]: May 17 19:23:47 cloud-init[2647]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
villasv commented 4 years ago

I can see that you have AllowHTTPAccessCIDR = 10.0.0.0/16, which means that you can only access the webserver if you're inside the VPC. If you don't have a VPN set up, I guess you can't do it. You can try to change it to 0.0.0.0/0 temporarily to see if you can access it. If you're inside the webserver shell, you can test it with a curl localhost:8080

Those logs are indeed troubling, though. But it doesn't provide sufficient details. IIRC you can see the setup logs in /var/log/cfn-init.log and /var/log/cfn-init-cmd.log, and consult the service logs with journalctl -u airflow-webserver.service.

ilyanoskov commented 4 years ago

@villasv this is what cfn-init.log gives me:

ws s3 sync s3://turbine-quickstart/quickstart-turbine-airflow/scripts /opt/turbine
chmod +x /opt/turbine/webserver.setup.sh
/opt/turbine/webserver.setup.sh
) failed
2020-05-17 20:48:53,692 [DEBUG] Command setup output: fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
chmod: cannot access '/opt/turbine/webserver.setup.sh': No such file or directory
/bin/sh: line 9: /opt/turbine/webserver.setup.sh: No such file or directory

2020-05-17 20:48:53,693 [ERROR] Error encountered during build of config: Command setup failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command setup failed
2020-05-17 20:48:53,694 [ERROR] -----------------------BUILD FAILED!------------------------
2020-05-17 20:48:53,696 [ERROR] Unhandled exception during build: Command setup failed
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 260, in build
    changes['commands'] = CommandTool().apply(self._config.commands)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/command_tool.py", line 117, in apply
    raise ToolError(u"Command %s failed" % name)
ToolError: Command setup failed

and cfn-init-cmd.log:

sh-4.2$ cat /var/log/cfn-init-cmd.log
2020-05-17 20:48:52,551 P2658 [INFO] ************************************************************
2020-05-17 20:48:52,552 P2658 [INFO] ConfigSet default
2020-05-17 20:48:52,552 P2658 [INFO] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2020-05-17 20:48:52,552 P2658 [INFO] Config config
2020-05-17 20:48:52,553 P2658 [INFO] ============================================================
2020-05-17 20:48:52,553 P2658 [INFO] Command setup
2020-05-17 20:48:53,691 P2658 [INFO] -----------------------Command Output-----------------------
2020-05-17 20:48:53,692 P2658 [INFO]    fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
2020-05-17 20:48:53,692 P2658 [INFO]    chmod: cannot access '/opt/turbine/webserver.setup.sh': No such file or directory
2020-05-17 20:48:53,692 P2658 [INFO]    /bin/sh: line 9: /opt/turbine/webserver.setup.sh: No such file or directory
2020-05-17 20:48:53,692 P2658 [INFO] ------------------------------------------------------------
2020-05-17 20:48:53,692 P2658 [ERROR] Exited with error code 127

journalctl:

sudo journalctl -u airflow-webserver.service
-- No entries --
villasv commented 4 years ago

Thanks @ilyanoskov, it looks like it's a permissioning problem in the s3 buckets as it all starts here:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

I'll check if the eu-central-1 has the right public access permissions.

ilyanoskov commented 4 years ago

@villasv thanks a lot! Please let me know when you do and I will try again with Cloudformation.

villasv commented 4 years ago

There you go. Not only the files had to be public, but there was a missing bucket policy for the bucket to make the s3 sync api work.

ilyanoskov commented 4 years ago

@villasv now I am able to connect to UI, thank you very much.

There are some errors I see in the session manager (journalctl) though:


-1-243.eu-central-1.compute.internal systemd[1]: Reloading.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Started cfn-hup.service.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Starting cfn-hup.service...
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Reloading.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Started Airflow webserver daemon.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Starting Airflow webserver daemon...
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Started airflow-confapply-agent.path.
May 18 11:36:50 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Starting airflow-confapply-agent.path.
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,166] {cli_action_loggers.py:107} WARNING - Failed to log action with (psycopg2.errors.Undef
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 1: INSERT INTO log (dttm, dag_id, task_id, event, execution_dat...
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (%(dttm)s, %(dag_id)s,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [parameters: {'dttm': datetime.datetime(2020, 5, 18, 11, 36, 51, 908287, tzinfo=<Timezone [UTC]>), 'dag_id': Non
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal rsyslogd[2442]: imjournal: journal reloaded... [v8.24.0-41.amzn2.2.1 try http://www.rsyslog.com/e/0 ]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: (Background on this error at: http://sqlalche.me/e/f405)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ____________       _____________
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ____    |__( )_________  __/__  /________      __
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,379] {__init__.py:51} INFO - Using executor CeleryExecutor
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,379] {dagbag.py:396} INFO - Filling up the DagBag from /airflow/dags
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [2020-05-18 11:36:52,422] {dagbag.py:239} ERROR - Failed to import: /usr/local/lib/python3.7/site-packages/airfl
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: psycopg2.errors.UndefinedTable: relation "slot_pool" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM slot_pool
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: The above exception was the direct cause of the following exception:
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 236, in process_file
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: m = imp.load_source(mod_name, filepath)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/lib64/python3.7/imp.py", line 171, in load_source
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: module = _load(spec)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap>", line 696, in _load
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap_external>", line 728, in exec_module
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/example_dags/example_subdag_operator.py", line 50, in <modu
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: dag=dag,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return func(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/decorators.py", line 98, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: result = func(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/operators/subdag_operator.py", line 77, in __init__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: .filter(Pool.pool == self.pool)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3375, in first
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ret = list(self[0:1])
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return list(res)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return self._execute_and_instances(context)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: result = conn.execute(querycontext.statement, self._params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return meth(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return connection._execute_clauseelement(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: distilled_params,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: e, statement, parameters, cursor, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy_exception, with_traceback=exc_info[2], from_=e
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: raise exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "slot_pool" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM slot_pool
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [SQL: SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: FROM slot_pool
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: WHERE slot_pool.slots = %(slots_1)s AND slot_pool.pool = %(pool_1)s
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LIMIT %(param_1)s]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [parameters: {'slots_1': 1, 'pool_1': 'default_pool', 'param_1': 1}]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: (Background on this error at: http://sqlalche.me/e/f405)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: psycopg2.errors.UndefinedTable: relation "connection" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM connection GROUP BY connection.conn_id
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: The above exception was the direct cause of the following exception:
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: Traceback (most recent call last):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/bin/airflow", line 37, in <module>
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: args.func(args)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return f(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 900, in webserver
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 232, in cached_app
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: app = create_app(config, testing)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 87, in create_app
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: from airflow.www import views
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2477, in <module>
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2568, in ChartModelView
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: 'conn_id': _connection_ids()
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return func(*args, **kwargs)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 2474, in _connection_ids
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: .group_by(Connection.conn_id))]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return self._execute_and_instances(context)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: result = conn.execute(querycontext.statement, self._params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return meth(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: return connection._execute_clauseelement(self, multiparams, params)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: distilled_params,
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: e, statement, parameters, cursor, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy_exception, with_traceback=exc_info[2], from_=e
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: raise exception
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor, statement, parameters, context
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: File "/usr/local/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: cursor.execute(statement, parameters)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "connection" does not exist
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: LINE 2: FROM connection GROUP BY connection.conn_id
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: ^
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: [SQL: SELECT connection.conn_id AS connection_conn_id
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: FROM connection GROUP BY connection.conn_id]
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal airflow[3239]: (Background on this error at: http://sqlalche.me/e/f405)
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: airflow-webserver.service: main process exited, code=exited, status=1/FAILURE
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Unit airflow-webserver.service entered failed state.
May 18 11:36:52 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: airflow-webserver.service failed.```
ilyanoskov commented 4 years ago

and this error as well:

May 18 11:37:02 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: + /opt/aws/bin/cfn-signal -e 0
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: Error: No WaitConditionHandle URL or logical resource id specified

May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: May 18 11:37:03 cloud-init[2647]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: May 18 11:37:03 cloud-init[2647]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: May 18 11:37:03 cloud-init[2647]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: ci-info: no authorized ssh keys fingerprints found for user ec2-user.
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal ec2[3436]:
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal ec2[3436]: 
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal cloud-init[2647]: Cloud-init v. 19.3-2.amzn2 finished at Mon, 18 May 2020 11:37:03 +0000. Datasource DataSourceEc2.  Up 166.91
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: cloud-final.service: main process exited, code=exited, status=1/FAILURE
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Failed to start Execute cloud user/final scripts.
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: Unit cloud-final.service entered failed state.
May 18 11:37:03 ip-10-0-1-243.eu-central-1.compute.internal systemd[1]: cloud-final.service failed.
villasv commented 4 years ago

There are some errors I see in the session manager (journalctl) though

That might happen temporarily, if the migrations are still ongoing. It shouldn't stay happening forever, though.

About your second logs paste:

Error: No WaitConditionHandle URL or logical resource id specified

This is an error indeed, but it's not relevant. There's no WaitConditionHandle in use by the stack at the moment, this is likely to be removed later on.

villasv commented 4 years ago

Let me know if the errors in the journal are persisting, or if your tasks are not being scheduled/executed properly.

ilyanoskov commented 4 years ago

ok, thank you very much! Will update you soon as I will have some test DAGs running.

You provided tremendous support man, the coffee is on me ;) I believe we can close this issue, now that I can connect to UI.