search

aws-samples / aws-concurrent-data-orchestration-pipeline-emr-livy

This code demonstrates the architecture featured on the AWS Big Data blog (https://aws.amazon.com/blogs/big-data/ ) which creates a concurrent data pipeline by using Amazon EMR and Apache Livy. This pipeline is orchestrated by Apache Airflow.
Apache License 2.0
76 stars 33 forks source link

Waiter ClusterRunning failed: Waiter encountered a terminal failure state #3

Open yoryicopo opened 5 years ago

yoryicopo commented 5 years ago

When I launch transform_movielens DAG, there's an error in the operation wait_for_cluster_completion. This is the output log /root/airflowlogs/transform_movielens/wait_for_cluster_completion/2018-11-20T11:10:11.781647+00:001.log:

Starting attempt 1 of 1
--------------------------------------------------------------------------------

[2018-11-20 11:10:18,607] {models.py:1569} INFO - Executing <Task(PythonOperator): wait_for_cluster_completion> on 2018-11-20T11:10:11.781647+00:00
[2018-11-20 11:10:18,608] {base_task_runner.py:124} INFO - Running: ['bash', '-c', u'airflow run transform_movielens wait_for_cluster_completion 2018-11-20T11:10:11.781647+00:00 --job_id 8 --raw -sd DAGS_
FOLDER/movielens_dag.py --cfg_path /tmp/tmpLbcGBC']
[2018-11-20 11:10:18,900] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion [2018-11-20 11:10:18,899] {settings.py:174} INFO - setting.configure_orm(): Using pool settings. pool_
size=5, pool_recycle=1800
[2018-11-20 11:10:19,044] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion [2018-11-20 11:10:19,043] {__init__.py:51} INFO - Using executor LocalExecutor
[2018-11-20 11:10:19,150] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion [2018-11-20 11:10:19,150] {models.py:258} INFO - Filling up the DagBag from /root/airflow/dags/moviele
ns_dag.py
[2018-11-20 11:10:19,210] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion [2018-11-20 11:10:19,210] {example_kubernetes_operator.py:54} WARNING - Could not import KubernetesPod
Operator: No module named kubernetes
[2018-11-20 11:10:19,211] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion [2018-11-20 11:10:19,210] {example_kubernetes_operator.py:55} WARNING - Install kubernetes dependencies with:     pip install airflow['kubernetes']
[2018-11-20 11:10:19,378] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion [2018-11-20 11:10:19,377] {cli.py:492} INFO - Running <TaskInstance: transform_movielens.wait_for_cluster_completion 2018-11-20T11:10:11.781647+00:00 [running]> on host ip-172-31-37-93.eu-west-1.compute.internal
[2018-11-20 11:10:49,574] {models.py:1736} ERROR - Waiter ClusterRunning failed: Waiter encountered a terminal failure state
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1633, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 95, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 100, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/root/airflow/dags/movielens_dag.py", line 32, in wait_for_completion
    emr.wait_for_cluster_creation(cluster_id)
  File "/root/airflow/dags/airflowlib/emr_lib.py", line 66, in wait_for_cluster_creation
    emr.get_waiter('cluster_running').wait(ClusterId=cluster_id)
  File "/usr/local/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/waiter.py", line 323, in wait
    last_response=response,
WaiterError: Waiter ClusterRunning failed: Waiter encountered a terminal failure state
[2018-11-20 11:10:49,576] {models.py:1764} INFO - Marking task as FAILED.
[2018-11-20 11:10:49,602] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion Traceback (most recent call last):
[2018-11-20 11:10:49,602] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/bin/airflow", line 32, in <module>
[2018-11-20 11:10:49,603] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     args.func(args)
[2018-11-20 11:10:49,603] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2018-11-20 11:10:49,603] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     return f(*args, **kwargs)
[2018-11-20 11:10:49,603] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 498, in run
[2018-11-20 11:10:49,603] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     _run(args, dag, ti)
[2018-11-20 11:10:49,603] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 402, in _run
[2018-11-20 11:10:49,604] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     pool=args.pool,
[2018-11-20 11:10:49,604] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in wrapper
[2018-11-20 11:10:49,604] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     return func(*args, **kwargs)
[2018-11-20 11:10:49,604] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1633, in _run_raw_task
[2018-11-20 11:10:49,604] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     result = task_copy.execute(context=context)
[2018-11-20 11:10:49,604] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 95, in execute
[2018-11-20 11:10:49,605] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     return_value = self.execute_callable()
[2018-11-20 11:10:49,605] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 100, in execute_callable
[2018-11-20 11:10:49,605] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     return self.python_callable(*self.op_args, **self.op_kwargs)
[2018-11-20 11:10:49,605] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/root/airflow/dags/movielens_dag.py", line 32, in wait_for_completion
[2018-11-20 11:10:49,605] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     emr.wait_for_cluster_creation(cluster_id)
[2018-11-20 11:10:49,605] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/root/airflow/dags/airflowlib/emr_lib.py", line 66, in wait_for_cluster_creation
[2018-11-20 11:10:49,606] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     emr.get_waiter('cluster_running').wait(ClusterId=cluster_id)
[2018-11-20 11:10:49,606] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait
[2018-11-20 11:10:49,606] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     Waiter.wait(self, **kwargs)
[2018-11-20 11:10:49,606] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion   File "/usr/local/lib/python2.7/site-packages/botocore/waiter.py", line 323, in wait
[2018-11-20 11:10:49,606] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion     last_response=response,
[2018-11-20 11:10:49,606] {base_task_runner.py:107} INFO - Job 8: Subtask wait_for_cluster_completion botocore.exceptions.WaiterError: Waiter ClusterRunning failed: Waiter encountered a terminal failure state
[2018-11-20 11:10:53,712] {logging_mixin.py:95} INFO - [2018-11-20 11:10:53,712] {jobs.py:2612} INFO - Task exited with return code 1
Eugst commented 5 years ago

+1

hc1wwilger commented 4 years ago

+1