search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.04k stars 14.29k forks source link

Attempting to read an xcom produced by KubernetesPodOperator results in UnicodeDecodeError #12813

Closed dmateusp closed 2 years ago

dmateusp commented 3 years ago

Apache Airflow version: 1.10.13

Kubernetes version (if you are using kubernetes) (use kubectl version): v1.15.11-eks

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-065dce", GitCommit:"065dcecfcd2a91bd68a17ee0b5e895088430bd05", GitTreeState:"clean", BuildDate:"2020-07-16T01:44:47Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Environment:

What happened:

xcom.pull(...) throws UnicodeDecodeError when the task that produced the xcom is KubernetesPodOperator:

    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line 161, in get_one
    return json.loads(result.value.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

(Full stack trace lower)

Note that this is happening across all of our DAGs when I try to migrate from 1.10.10 to 1.10.13

How to reproduce it:

Here are 2 tasks that should reproduce it (that is part of a DAG I use to test K8s features because making an update internally):

    bash_echo_1 = KubernetesPodOperator(
        task_id="bash_echo_1",
        image="bash:4.4",
        name="bash-echo-1",  # kubernetes pod names do not accept '_'
        cmds=[
            "bash",
            "-c",
            (
                "mkdir -p /airflow/xcom "
                '&& echo \'{"key1":"value1", "key2": "value2"}\' > /airflow/xcom/return.json'
            ),
        ],  # that's how xcom works for KubernetesPodOperator
        # on_failure_callback=alert_opsgenie,  # uncomment to test opgenie
        do_xcom_push=True,
    )  # this needs to be set to true for `/airflow/xcom/return.json` to be pushed as an xcom object

    bash_echo_2 = KubernetesPodOperator(
        task_id="bash_echo_2",
        name="bash-echo-2",  # kubernetes pod names do not accept '_'
        image="bash:4.4",
        arguments=[
            "echo",
            'key1 was: {{ ti.xcom_pull("bash_echo_1")["key1"] }}',
            ',key2 was: {{ ti.xcom_pull("bash_echo_1")["key2"] }}',
            ',the entire object was: {{ ti.xcom_pull("bash_echo_1") }}',
        ],
    )

    bash_echo_1 >> bash_echo_2
stack trace ``` Process DagFileProcessor324119-Process: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 159, in _run_file_processor pickle_dags) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1620, in process_file self._process_dags(dagbag, dags, ti_keys_to_schedule) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1299, in _process_dags self._process_task_instances(dag, tis_out) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 793, in _process_task_instances ready_tis = run.update_state(session=session) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 281, in update_state ready_tis, changed_tis = self._get_ready_tis(scheduleable_tasks, finished_tasks, session) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 340, in _get_ready_tis session=session): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 659, in are_dependencies_met session=session): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses dep_context): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses for dep_status in self._get_dep_statuses(ti, session, dep_context): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/not_previously_skipped_dep.py", line 58, in _get_dep_statuses task_ids=parent.task_id, key=XCOM_SKIPMIXIN_KEY File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1564, in xcom_pull return pull_fn(task_id=task_ids) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/xcom.py", line 161, in get_one return json.loads(result.value.decode('UTF-8')) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte ```

Note that bash_echo_2 simply does not get scheduled, the stack trace here comes from the scheduler Pod (we run Airflow on KubernetesExecutor)

boring-cyborg[bot] commented 3 years ago

Thanks for opening your first issue here! Be sure to follow the issue template!

potiuk commented 3 years ago

@dimberman ? Does it ring a bell ?

blessymoses commented 3 years ago

This issue also results in BranchPythonOperator not working with KubernetesExecutor.

Apache Airflow version: 1.10.12

What happened: BranchPythonOperator task succeeds. But it does not schedule the task in the followed branch. Scheduler throws UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Stack Trace: Process DagFileProcessor4425-Process: Traceback (most recent call last): File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, self._kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 159, in _run_file_processor pickle_dags) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1637, in process_file self._process_dags(dagbag, dags, ti_keys_to_schedule) File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1316, in _process_dags self._process_task_instances(dag, tis_out) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(args, kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 793, in _process_task_instances run.update_state(session=session) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper return func(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 296, in update_state session=session) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 70, in wrapper return func(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 659, in are_dependencies_met session=session): File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses dep_context): File "/usr/local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses for dep_status in self._get_dep_statuses(ti, session, dep_context): File "/usr/local/lib/python3.7/site-packages/airflow/ti_deps/deps/not_previously_skipped_dep.py", line 58, in _get_dep_statuses task_ids=parent.task_id, key=XCOM_SKIPMIXIN_KEY File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1564, in xcom_pull return pull_fn(task_id=task_ids) File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(args, kwargs) File "/usr/local/lib/python3.7/site-packages/airflow/models/xcom.py", line 165, in get_one return json.loads(result.value.decode('UTF-8')) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

kaxil commented 3 years ago

Does it happen on 2.0.0 too ?

dimberman commented 3 years ago

@dmateusp have you tried this on 2.0?

dmateusp commented 3 years ago

sorry folks, we didn't attempt to migrate to 2.0 yet!

michaelosthege commented 3 years ago

We're getting the same error in Airflow 2.0.1 and it happens with xcom produced by the PythonOperator too.

Tasks that try to xcom_pull upstream results that were created before the 1.0.14 → 2.0.1 migration break (see below). Pulling upstream XCom results that were created after the 2.0.1 upgrade works just fine.

[2021-04-17 14:28:35,344] {taskinstance.py:1455} ERROR - 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "/opt/conda/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
    return_value = self.execute_callable()
  File "/opt/conda/lib/python3.7/site-packages/airflow/operators/python.py", line 128, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/flow.py", line 91, in _run
    upstream_result = task_instance.xcom_pull(task_ids=[taskid])[0]
  File "/opt/conda/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1960, in xcom_pull
    for result in query.with_entities(XCom.task_id, XCom.value)
  File "/opt/conda/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1960, in <dictcomp>
    for result in query.with_entities(XCom.task_id, XCom.value)
  File "/opt/conda/lib/python3.7/site-packages/airflow/models/xcom.py", line 255, in deserialize_value
    return json.loads(result.value.decode('UTF-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

Interestingly the XCom data shows up just fine on the webserver: grafik

dimberman commented 3 years ago

@Dr-Denzy @ephraimbuddy would either of you be interested in this ticket?

Dr-Denzy commented 3 years ago

Let's collaborate on this @ephraimbuddy

dimberman commented 3 years ago

Hi @Dr-Denzy @ephraimbuddy any update on this?

ephraimbuddy commented 3 years ago

I was not able to reproduce this on master. Here's my dag:

DAG code ```python from airflow import DAG from datetime import datetime from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator default_args = { 'owner': 'Airflow'} with DAG('dag_pod_operatorxcom', start_date=datetime(2019, 3, 12), schedule_interval='@once', default_args=default_args, tags=["core", "http"]) as dag: bash_echo_1 = KubernetesPodOperator( in_cluster=True, namespace='airflow', task_id="bash_echo_1", image="bash:4.4", name="bash-echo-1", # kubernetes pod names do not accept '_' cmds=[ "bash", "-c", ( "mkdir -p /airflow/xcom " '&& echo \'{"key1":"value1", "key2": "value2"}\' > /airflow/xcom/return.json' ), ], # that's how xcom works for KubernetesPodOperator # on_failure_callback=alert_opsgenie, # uncomment to test opgenie do_xcom_push=True, ) # this needs to be set to true for `/airflow/xcom/return.json` to be pushed as an xcom object bash_echo_2 = KubernetesPodOperator( in_cluster=True, namespace='airflow', task_id="bash_echo_2", name="bash-echo-2", # kubernetes pod names do not accept '_' image="bash:4.4", arguments=[ "echo", 'key1 was: {{ ti.xcom_pull("bash_echo_1")["key1"] }}', ',key2 was: {{ ti.xcom_pull("bash_echo_1")["key2"] }}', ',the entire object was: {{ ti.xcom_pull("bash_echo_1") }}', ], ) bash_echo_1 >> bash_echo_2 ```

Here's logs for first task:

Task bash_echo_1 log ```log *** Reading remote log from s3://ephraim-airflowtest/logs/dag_pod_operatorxcom/bash_echo_1/2021-04-30T10:05:08.123775+00:00/1.log. [2021-04-30 10:05:12,625] {base_task_runner.py:62} DEBUG - Planning to run as the user [2021-04-30 10:05:12,627] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:12,632] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - dependency 'Task Instance Not Running' PASSED: True, Task is not in running state. [2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks. [2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying. [2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set. [2021-04-30 10:05:12,633] {taskinstance.py:892} DEBUG - dependency 'Task Instance State' PASSED: True, Task state queued was valid. [2021-04-30 10:05:12,636] {taskinstance.py:877} INFO - Dependencies all met for [2021-04-30 10:05:12,636] {taskinstance.py:892} DEBUG - dependency 'Trigger Rule' PASSED: True, The task instance did not have any upstream tasks. [2021-04-30 10:05:12,636] {taskinstance.py:892} DEBUG - dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying. [2021-04-30 10:05:12,639] {taskinstance.py:892} DEBUG - dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set. [2021-04-30 10:05:12,642] {taskinstance.py:892} DEBUG - dependency 'Pool Slots Available' PASSED: True, ('There are enough open slots in %s to execute the task', 'default_pool') [2021-04-30 10:05:12,642] {taskinstance.py:892} DEBUG - dependency 'Task Concurrency' PASSED: True, Task concurrency is not set. [2021-04-30 10:05:12,642] {taskinstance.py:877} INFO - Dependencies all met for [2021-04-30 10:05:12,642] {taskinstance.py:1068} INFO - -------------------------------------------------------------------------------- [2021-04-30 10:05:12,642] {taskinstance.py:1069} INFO - Starting attempt 1 of 1 [2021-04-30 10:05:12,642] {taskinstance.py:1070} INFO - -------------------------------------------------------------------------------- [2021-04-30 10:05:12,651] {taskinstance.py:1089} INFO - Executing on 2021-04-30T10:05:08.123775+00:00 [2021-04-30 10:05:12,653] {standard_task_runner.py:52} INFO - Started process 31 to run task [2021-04-30 10:05:12,655] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'dag_pod_operatorxcom', 'bash_echo_1', '2021-04-30T10:05:08.123775+00:00', '--job-id', '4', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/podxcompod.py', '--cfg-path', '/tmp/tmp50ostpx4', '--error-file', '/tmp/tmpjti1u7br'] [2021-04-30 10:05:12,655] {standard_task_runner.py:77} INFO - Job 4: Subtask bash_echo_1 [2021-04-30 10:05:12,656] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [] [2021-04-30 10:05:12,663] {settings.py:208} DEBUG - Setting up DB connection pool (PID 31) [2021-04-30 10:05:12,664] {settings.py:241} DEBUG - settings.prepare_engine_args(): Using NullPool [2021-04-30 10:05:12,665] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:12,675] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:12,688] {logging_mixin.py:104} INFO - Running on host dagpodoperatorxcombashecho1.c66682c5cfd84c5a92dfa3ed0e5efe5a [2021-04-30 10:05:12,688] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:12,694] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:12,696] {taskinstance.py:658} DEBUG - Clearing XCom data [2021-04-30 10:05:12,701] {taskinstance.py:665} DEBUG - XCom data cleared [2021-04-30 10:05:12,727] {taskinstance.py:1282} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=Airflow AIRFLOW_CTX_DAG_ID=dag_pod_operatorxcom AIRFLOW_CTX_TASK_ID=bash_echo_1 AIRFLOW_CTX_EXECUTION_DATE=2021-04-30T10:05:08.123775+00:00 AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-30T10:05:08.123775+00:00 [2021-04-30 10:05:12,727] {__init__.py:146} DEBUG - Preparing lineage inlets and outlets [2021-04-30 10:05:12,727] {__init__.py:190} DEBUG - inlets: [], outlets: [] [2021-04-30 10:05:12,727] {kubernetes_pod.py:434} DEBUG - Creating pod for KubernetesPodOperator task bash_echo_1 [2021-04-30 10:05:12,728] {kubernetes_pod.py:491} DEBUG - Adding xcom sidecar to task bash_echo_1 [2021-04-30 10:05:12,735] {rest.py:228} DEBUG - response body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"8373"},"items":[]} [2021-04-30 10:05:12,736] {kubernetes_pod.py:367} INFO - creating pod with labels {'dag_id': 'dag_pod_operatorxcom', 'task_id': 'bash_echo_1', 'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc', 'try_number': '1'} and launcher [2021-04-30 10:05:12,736] {kubernetes_pod.py:504} DEBUG - Adding KubernetesPodOperator labels to pod before launch for task bash_echo_1 [2021-04-30 10:05:12,746] {kubernetes_pod.py:518} DEBUG - Starting pod: api_version: v1 kind: Pod metadata: annotations: {} cluster_name: null creation_timestamp: null deletion_grace_period_seconds: null deletion_timestamp: null finalizers: null generate_name: null generation: null initializers: null labels: airflow_version: 2.1.0.dev0 dag_id: dag_pod_operatorxcom execution_date: 2021-04-30T100508.1237750000-f80bbc1dc kubernetes_pod_operator: 'True' task_id: bash_echo_1 try_number: '1' managed_fields: null name: bash-echo-1.741ec5fc537942479902c03db8bae9c9 namespace: airflow owner_references: null resource_version: null self_link: null uid: null spec: active_deadline_seconds: null affinity: node_affinity: null pod_affinity: null pod_anti_affinity: null automount_service_account_token: null containers: - args: [] command: - bash - -c - 'mkdir -p /airflow/xcom && echo ''{"key1":"value1", "key2": "value2"}'' > /airflow/xcom/return.json' env: [] env_from: [] image: bash:4.4 image_pull_policy: IfNotPresent lifecycle: null liveness_probe: null name: base ports: [] readiness_probe: null resources: {} security_context: null stdin: null stdin_once: null termination_message_path: null termination_message_policy: null tty: null volume_devices: null volume_mounts: - mount_path: /airflow/xcom mount_propagation: null name: xcom read_only: null sub_path: null sub_path_expr: null working_dir: null - args: null command: - sh - -c - trap "exit 0" INT; while true; do sleep 1; done; env: null env_from: null image: alpine image_pull_policy: null lifecycle: null liveness_probe: null name: airflow-xcom-sidecar ports: null readiness_probe: null resources: limits: null requests: cpu: 1m security_context: null stdin: null stdin_once: null termination_message_path: null termination_message_policy: null tty: null volume_devices: null volume_mounts: - mount_path: /airflow/xcom mount_propagation: null name: xcom read_only: null sub_path: null sub_path_expr: null working_dir: null dns_config: null dns_policy: null enable_service_links: null host_aliases: null host_ipc: null host_network: false host_pid: null hostname: null image_pull_secrets: [] init_containers: [] node_name: null node_selector: {} preemption_policy: null priority: null priority_class_name: null readiness_gates: null restart_policy: Never runtime_class_name: null scheduler_name: null security_context: {} service_account: null service_account_name: default share_process_namespace: null subdomain: null termination_grace_period_seconds: null tolerations: [] volumes: - aws_elastic_block_store: null azure_disk: null azure_file: null cephfs: null cinder: null config_map: null csi: null downward_api: null empty_dir: medium: null size_limit: null fc: null flex_volume: null flocker: null gce_persistent_disk: null git_repo: null glusterfs: null host_path: null iscsi: null name: xcom nfs: null persistent_volume_claim: null photon_persistent_disk: null portworx_volume: null projected: null quobyte: null rbd: null scale_io: null secret: null storageos: null vsphere_volume: null status: null [2021-04-30 10:05:12,747] {pod_launcher.py:86} DEBUG - Pod Creation Request: { "apiVersion": "v1", "kind": "Pod", "metadata": { "annotations": {}, "labels": { "dag_id": "dag_pod_operatorxcom", "task_id": "bash_echo_1", "execution_date": "2021-04-30T100508.1237750000-f80bbc1dc", "try_number": "1", "airflow_version": "2.1.0.dev0", "kubernetes_pod_operator": "True" }, "name": "bash-echo-1.741ec5fc537942479902c03db8bae9c9", "namespace": "airflow" }, "spec": { "affinity": {}, "containers": [ { "args": [], "command": [ "bash", "-c", "mkdir -p /airflow/xcom && echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' > /airflow/xcom/return.json" ], "env": [], "envFrom": [], "image": "bash:4.4", "imagePullPolicy": "IfNotPresent", "name": "base", "ports": [], "resources": {}, "volumeMounts": [ { "mountPath": "/airflow/xcom", "name": "xcom" } ] }, { "command": [ "sh", "-c", "trap \"exit 0\" INT; while true; do sleep 1; done;" ], "image": "alpine", "name": "airflow-xcom-sidecar", "resources": { "requests": { "cpu": "1m" } }, "volumeMounts": [ { "mountPath": "/airflow/xcom", "name": "xcom" } ] } ], "hostNetwork": false, "imagePullSecrets": [], "initContainers": [], "nodeSelector": {}, "restartPolicy": "Never", "securityContext": {}, "serviceAccountName": "default", "tolerations": [], "volumes": [ { "emptyDir": {}, "name": "xcom" } ] } } [2021-04-30 10:05:12,750] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8374","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","qosClass":"Burstable"}} [2021-04-30 10:05:12,751] {pod_launcher.py:91} DEBUG - Pod Creation Response: {'api_version': 'v1', 'kind': 'Pod', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2021, 4, 30, 10, 5, 12, tzinfo=tzlocal()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': {'airflow_version': '2.1.0.dev0', 'dag_id': 'dag_pod_operatorxcom', 'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc', 'kubernetes_pod_operator': 'True', 'task_id': 'bash_echo_1', 'try_number': '1'}, 'managed_fields': [{'api_version': 'v1', 'fields': None, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': datetime.datetime(2021, 4, 30, 10, 5, 12, tzinfo=tzlocal())}], 'name': 'bash-echo-1.741ec5fc537942479902c03db8bae9c9', 'namespace': 'airflow', 'owner_references': None, 'resource_version': '8374', 'self_link': None, 'uid': '688b6de8-e99b-4caa-894a-750480ead1c8'}, 'spec': {'active_deadline_seconds': None, 'affinity': {'node_affinity': None, 'pod_affinity': None, 'pod_anti_affinity': None}, 'automount_service_account_token': None, 'containers': [{'args': None, 'command': ['bash', '-c', 'mkdir -p /airflow/xcom && echo ' '\'{"key1":"value1", "key2": "value2"}\' ' '> /airflow/xcom/return.json'], 'env': None, 'env_from': None, 'image': 'bash:4.4', 'image_pull_policy': 'IfNotPresent', 'lifecycle': None, 'liveness_probe': None, 'name': 'base', 'ports': None, 'readiness_probe': None, 'resources': {'limits': None, 'requests': None}, 'security_context': None, 'stdin': None, 'stdin_once': None, 'termination_message_path': '/dev/termination-log', 'termination_message_policy': 'File', 'tty': None, 'volume_devices': None, 'volume_mounts': [{'mount_path': '/airflow/xcom', 'mount_propagation': None, 'name': 'xcom', 'read_only': None, 'sub_path': None, 'sub_path_expr': None}, {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount', 'mount_propagation': None, 'name': 'default-token-f546n', 'read_only': True, 'sub_path': None, 'sub_path_expr': None}], 'working_dir': None}, {'args': None, 'command': ['sh', '-c', 'trap "exit 0" INT; while true; do sleep ' '1; done;'], 'env': None, 'env_from': None, 'image': 'alpine', 'image_pull_policy': 'Always', 'lifecycle': None, 'liveness_probe': None, 'name': 'airflow-xcom-sidecar', 'ports': None, 'readiness_probe': None, 'resources': {'limits': None, 'requests': {'cpu': '1m'}}, 'security_context': None, 'stdin': None, 'stdin_once': None, 'termination_message_path': '/dev/termination-log', 'termination_message_policy': 'File', 'tty': None, 'volume_devices': None, 'volume_mounts': [{'mount_path': '/airflow/xcom', 'mount_propagation': None, 'name': 'xcom', 'read_only': None, 'sub_path': None, 'sub_path_expr': None}, {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount', 'mount_propagation': None, 'name': 'default-token-f546n', 'read_only': True, 'sub_path': None, 'sub_path_expr': None}], 'working_dir': None}], 'dns_config': None, 'dns_policy': 'ClusterFirst', 'enable_service_links': True, 'host_aliases': None, 'host_ipc': None, 'host_network': None, 'host_pid': None, 'hostname': None, 'image_pull_secrets': None, 'init_containers': None, 'node_name': None, 'node_selector': None, 'preemption_policy': 'PreemptLowerPriority', 'priority': 0, 'priority_class_name': None, 'readiness_gates': None, 'restart_policy': 'Never', 'runtime_class_name': None, 'scheduler_name': 'default-scheduler', 'security_context': {'fs_group': None, 'run_as_group': None, 'run_as_non_root': None, 'run_as_user': None, 'se_linux_options': None, 'supplemental_groups': None, 'sysctls': None, 'windows_options': None}, 'service_account': 'default', 'service_account_name': 'default', 'share_process_namespace': None, 'subdomain': None, 'termination_grace_period_seconds': 30, 'tolerations': [{'effect': 'NoExecute', 'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'toleration_seconds': 300, 'value': None}, {'effect': 'NoExecute', 'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'toleration_seconds': 300, 'value': None}], 'volumes': [{'aws_elastic_block_store': None, 'azure_disk': None, 'azure_file': None, 'cephfs': None, 'cinder': None, 'config_map': None, 'csi': None, 'downward_api': None, 'empty_dir': {'medium': None, 'size_limit': None}, 'fc': None, 'flex_volume': None, 'flocker': None, 'gce_persistent_disk': None, 'git_repo': None, 'glusterfs': None, 'host_path': None, 'iscsi': None, 'name': 'xcom', 'nfs': None, 'persistent_volume_claim': None, 'photon_persistent_disk': None, 'portworx_volume': None, 'projected': None, 'quobyte': None, 'rbd': None, 'scale_io': None, 'secret': None, 'storageos': None, 'vsphere_volume': None}, {'aws_elastic_block_store': None, 'azure_disk': None, 'azure_file': None, 'cephfs': None, 'cinder': None, 'config_map': None, 'csi': None, 'downward_api': None, 'empty_dir': None, 'fc': None, 'flex_volume': None, 'flocker': None, 'gce_persistent_disk': None, 'git_repo': None, 'glusterfs': None, 'host_path': None, 'iscsi': None, 'name': 'default-token-f546n', 'nfs': None, 'persistent_volume_claim': None, 'photon_persistent_disk': None, 'portworx_volume': None, 'projected': None, 'quobyte': None, 'rbd': None, 'scale_io': None, 'secret': {'default_mode': 420, 'items': None, 'optional': None, 'secret_name': 'default-token-f546n'}, 'storageos': None, 'vsphere_volume': None}]}, 'status': {'conditions': None, 'container_statuses': None, 'host_ip': None, 'init_container_statuses': None, 'message': None, 'nominated_node_name': None, 'phase': 'Pending', 'pod_ip': None, 'qos_class': 'Burstable', 'reason': None, 'start_time': None}} [2021-04-30 10:05:12,761] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8375","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"qosClass":"Burstable"}} [2021-04-30 10:05:12,762] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending [2021-04-30 10:05:12,762] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9 [2021-04-30 10:05:13,767] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8377","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"alpine","imageID":"","started":false},{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:13,768] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending [2021-04-30 10:05:13,768] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9 [2021-04-30 10:05:14,772] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8377","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"alpine","imageID":"","started":false},{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:14,773] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending [2021-04-30 10:05:14,773] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9 [2021-04-30 10:05:15,782] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8377","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base airflow-xcom-sidecar]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"alpine","imageID":"","started":false},{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:15,783] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Pending [2021-04-30 10:05:15,784] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-1.741ec5fc537942479902c03db8bae9c9 [2021-04-30 10:05:16,787] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:16,789] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Running [2021-04-30 10:05:17,693] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:17,699] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:17,700] {base_job.py:219} DEBUG - [heartbeat] [2021-04-30 10:05:17,801] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:17,804] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:17,816] {pod_launcher.py:282} INFO - Running command... cat /airflow/xcom/return.json [2021-04-30 10:05:17,861] {pod_launcher.py:282} INFO - Running command... kill -s SIGINT 1 [2021-04-30 10:05:18,002] {pod_launcher.py:165} INFO - {"key1":"value1", "key2": "value2"} [2021-04-30 10:05:18,004] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8390","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"running":{"startedAt":"2021-04-30T10:05:15Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":true},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:18,006] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Running [2021-04-30 10:05:18,006] {pod_launcher.py:168} INFO - Pod bash-echo-1.741ec5fc537942479902c03db8bae9c9 has state running [2021-04-30 10:05:20,012] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8397","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:18Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:15Z","finishedAt":"2021-04-30T10:05:17Z","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":false},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:20,015] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Succeeded [2021-04-30 10:05:20,015] {pod_launcher.py:302} INFO - Event with job id bash-echo-1.741ec5fc537942479902c03db8bae9c9 Succeeded [2021-04-30 10:05:20,018] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-1.741ec5fc537942479902c03db8bae9c9","namespace":"airflow","uid":"688b6de8-e99b-4caa-894a-750480ead1c8","resourceVersion":"8397","creationTimestamp":"2021-04-30T10:05:12Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_1","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"airflow-xcom-sidecar\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}},"k:{\"name\":\"base\"}":{".":{},"f:command":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/airflow/xcom\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"xcom\"}":{".":{},"f:emptyDir":{},"f:name":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:18Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.45\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"xcom","emptyDir":{}},{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","command":["bash","-c","mkdir -p /airflow/xcom \u0026\u0026 echo '{\"key1\":\"value1\", \"key2\": \"value2\"}' \u003e /airflow/xcom/return.json"],"resources":{},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"},{"name":"airflow-xcom-sidecar","image":"alpine","command":["sh","-c","trap \"exit 0\" INT; while true; do sleep 1; done;"],"resources":{"requests":{"cpu":"1m"}},"volumeMounts":[{"name":"xcom","mountPath":"/airflow/xcom"},{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:12Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.45","podIPs":[{"ip":"10.244.1.45"}],"startTime":"2021-04-30T10:05:12Z","containerStatuses":[{"name":"airflow-xcom-sidecar","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:15Z","finishedAt":"2021-04-30T10:05:17Z","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/alpine:latest","imageID":"docker.io/library/alpine@sha256:69e70a79f2d41ab5d637de98c1e0b055206ba40a8145e7bddb55ccc04e13cf8f","containerID":"containerd://56df3aa01b943b3d5f842f5a08e1ab1c87954f6cf2bc65be8dc189f216d67649","started":false},{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:13Z","finishedAt":"2021-04-30T10:05:13Z","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://949dee18308f015a66ccbc9a08606bb6a168841371504ae53b53503eee100e0b","started":false}],"qosClass":"Burstable"}} [2021-04-30 10:05:20,020] {pod_launcher.py:189} INFO - Event: bash-echo-1.741ec5fc537942479902c03db8bae9c9 had an event of type Succeeded [2021-04-30 10:05:20,021] {pod_launcher.py:302} INFO - Event with job id bash-echo-1.741ec5fc537942479902c03db8bae9c9 Succeeded [2021-04-30 10:05:20,032] {__init__.py:107} DEBUG - Lineage called with inlets: [], outlets: [] [2021-04-30 10:05:20,033] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:20,039] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:20,040] {taskinstance.py:1191} INFO - Marking task as SUCCESS. dag_id=dag_pod_operatorxcom, task_id=bash_echo_1, execution_date=20210430T100508, start_date=20210430T100512, end_date=20210430T100520 [2021-04-30 10:05:20,040] {taskinstance.py:1887} DEBUG - Task Duration set to 7.404366 [2021-04-30 10:05:20,060] {dagrun.py:491} DEBUG - number of tis tasks for : 2 task(s) [2021-04-30 10:05:20,060] {dagrun.py:506} DEBUG - number of scheduleable tasks for : 1 task(s) [2021-04-30 10:05:20,061] {taskinstance.py:892} DEBUG - dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set. [2021-04-30 10:05:20,061] {taskinstance.py:892} DEBUG - dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying. [2021-04-30 10:05:20,061] {taskinstance.py:877} DEBUG - Dependencies all met for [2021-04-30 10:05:20,063] {taskinstance.py:1245} INFO - 1 downstream tasks scheduled from follow-on schedule check [2021-04-30 10:05:20,063] {cli_action_loggers.py:84} DEBUG - Calling callbacks: [] [2021-04-30 10:05:20,077] {local_task_job.py:152} INFO - Task exited with return code 0 [2021-04-30 10:05:20,077] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:20,084] {taskinstance.py:630} DEBUG - Refreshed TaskInstance ```

And logs for the second task:

bash echo 2 task ```log *** Reading remote log from s3://ephraim-airflowtest/logs/dag_pod_operatorxcom/bash_echo_2/2021-04-30T10:05:08.123775+00:00/1.log. [2021-04-30 10:05:24,707] {base_task_runner.py:62} DEBUG - Planning to run as the user [2021-04-30 10:05:24,708] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:24,714] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:24,715] {taskinstance.py:892} DEBUG - dependency 'Task Instance Not Running' PASSED: True, Task is not in running state. [2021-04-30 10:05:24,715] {taskinstance.py:892} DEBUG - dependency 'Task Instance State' PASSED: True, Task state queued was valid. [2021-04-30 10:05:24,715] {taskinstance.py:892} DEBUG - dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying. [2021-04-30 10:05:24,718] {taskinstance.py:892} DEBUG - dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set. [2021-04-30 10:05:24,718] {taskinstance.py:877} INFO - Dependencies all met for [2021-04-30 10:05:24,718] {taskinstance.py:892} DEBUG - dependency 'Task Concurrency' PASSED: True, Task concurrency is not set. [2021-04-30 10:05:24,721] {taskinstance.py:892} DEBUG - dependency 'Pool Slots Available' PASSED: True, ('There are enough open slots in %s to execute the task', 'default_pool') [2021-04-30 10:05:24,721] {taskinstance.py:892} DEBUG - dependency 'Not In Retry Period' PASSED: True, The task instance was not marked for retrying. [2021-04-30 10:05:24,721] {taskinstance.py:892} DEBUG - dependency 'Previous Dagrun State' PASSED: True, The task did not have depends_on_past set. [2021-04-30 10:05:24,729] {taskinstance.py:877} INFO - Dependencies all met for [2021-04-30 10:05:24,729] {taskinstance.py:1068} INFO - -------------------------------------------------------------------------------- [2021-04-30 10:05:24,729] {taskinstance.py:1069} INFO - Starting attempt 1 of 1 [2021-04-30 10:05:24,729] {taskinstance.py:1070} INFO - -------------------------------------------------------------------------------- [2021-04-30 10:05:24,734] {taskinstance.py:1089} INFO - Executing on 2021-04-30T10:05:08.123775+00:00 [2021-04-30 10:05:24,736] {standard_task_runner.py:52} INFO - Started process 31 to run task [2021-04-30 10:05:24,739] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'dag_pod_operatorxcom', 'bash_echo_2', '2021-04-30T10:05:08.123775+00:00', '--job-id', '5', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/podxcompod.py', '--cfg-path', '/tmp/tmp8kwa7j0v', '--error-file', '/tmp/tmpgcegf545'] [2021-04-30 10:05:24,739] {standard_task_runner.py:77} INFO - Job 5: Subtask bash_echo_2 [2021-04-30 10:05:24,740] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [] [2021-04-30 10:05:24,747] {settings.py:208} DEBUG - Setting up DB connection pool (PID 31) [2021-04-30 10:05:24,747] {settings.py:241} DEBUG - settings.prepare_engine_args(): Using NullPool [2021-04-30 10:05:24,749] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:24,759] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:24,771] {logging_mixin.py:104} INFO - Running on host dagpodoperatorxcombashecho2.0b7976cfc058439da85e87d672944d13 [2021-04-30 10:05:24,771] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:24,777] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:24,779] {taskinstance.py:658} DEBUG - Clearing XCom data [2021-04-30 10:05:24,784] {taskinstance.py:665} DEBUG - XCom data cleared [2021-04-30 10:05:24,827] {taskinstance.py:1282} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=Airflow AIRFLOW_CTX_DAG_ID=dag_pod_operatorxcom AIRFLOW_CTX_TASK_ID=bash_echo_2 AIRFLOW_CTX_EXECUTION_DATE=2021-04-30T10:05:08.123775+00:00 AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-30T10:05:08.123775+00:00 [2021-04-30 10:05:24,827] {__init__.py:146} DEBUG - Preparing lineage inlets and outlets [2021-04-30 10:05:24,827] {__init__.py:190} DEBUG - inlets: [], outlets: [] [2021-04-30 10:05:24,828] {kubernetes_pod.py:434} DEBUG - Creating pod for KubernetesPodOperator task bash_echo_2 [2021-04-30 10:05:24,835] {rest.py:228} DEBUG - response body: {"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"8415"},"items":[]} [2021-04-30 10:05:24,836] {kubernetes_pod.py:367} INFO - creating pod with labels {'dag_id': 'dag_pod_operatorxcom', 'task_id': 'bash_echo_2', 'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc', 'try_number': '1'} and launcher [2021-04-30 10:05:24,836] {kubernetes_pod.py:504} DEBUG - Adding KubernetesPodOperator labels to pod before launch for task bash_echo_2 [2021-04-30 10:05:24,841] {kubernetes_pod.py:518} DEBUG - Starting pod: api_version: v1 kind: Pod metadata: annotations: {} cluster_name: null creation_timestamp: null deletion_grace_period_seconds: null deletion_timestamp: null finalizers: null generate_name: null generation: null initializers: null labels: airflow_version: 2.1.0.dev0 dag_id: dag_pod_operatorxcom execution_date: 2021-04-30T100508.1237750000-f80bbc1dc kubernetes_pod_operator: 'True' task_id: bash_echo_2 try_number: '1' managed_fields: null name: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 namespace: airflow owner_references: null resource_version: null self_link: null uid: null spec: active_deadline_seconds: null affinity: node_affinity: null pod_affinity: null pod_anti_affinity: null automount_service_account_token: null containers: - args: - echo - 'key1 was: value1' - ',key2 was: value2' - ',the entire object was: {''key1'': ''value1'', ''key2'': ''value2''}' command: [] env: [] env_from: [] image: bash:4.4 image_pull_policy: IfNotPresent lifecycle: null liveness_probe: null name: base ports: [] readiness_probe: null resources: {} security_context: null stdin: null stdin_once: null termination_message_path: null termination_message_policy: null tty: null volume_devices: null volume_mounts: [] working_dir: null dns_config: null dns_policy: null enable_service_links: null host_aliases: null host_ipc: null host_network: false host_pid: null hostname: null image_pull_secrets: [] init_containers: [] node_name: null node_selector: {} preemption_policy: null priority: null priority_class_name: null readiness_gates: null restart_policy: Never runtime_class_name: null scheduler_name: null security_context: {} service_account: null service_account_name: default share_process_namespace: null subdomain: null termination_grace_period_seconds: null tolerations: [] volumes: [] status: null [2021-04-30 10:05:24,842] {pod_launcher.py:86} DEBUG - Pod Creation Request: { "apiVersion": "v1", "kind": "Pod", "metadata": { "annotations": {}, "labels": { "dag_id": "dag_pod_operatorxcom", "task_id": "bash_echo_2", "execution_date": "2021-04-30T100508.1237750000-f80bbc1dc", "try_number": "1", "airflow_version": "2.1.0.dev0", "kubernetes_pod_operator": "True" }, "name": "bash-echo-2.c3a98f36477a43f598c786a3cb2efc55", "namespace": "airflow" }, "spec": { "affinity": {}, "containers": [ { "args": [ "echo", "key1 was: value1", ",key2 was: value2", ",the entire object was: {'key1': 'value1', 'key2': 'value2'}" ], "command": [], "env": [], "envFrom": [], "image": "bash:4.4", "imagePullPolicy": "IfNotPresent", "name": "base", "ports": [], "resources": {}, "volumeMounts": [] } ], "hostNetwork": false, "imagePullSecrets": [], "initContainers": [], "nodeSelector": {}, "restartPolicy": "Never", "securityContext": {}, "serviceAccountName": "default", "tolerations": [], "volumes": [] } } [2021-04-30 10:05:24,844] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8416","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","qosClass":"BestEffort"}} [2021-04-30 10:05:24,846] {pod_launcher.py:91} DEBUG - Pod Creation Response: {'api_version': 'v1', 'kind': 'Pod', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': datetime.datetime(2021, 4, 30, 10, 5, 24, tzinfo=tzlocal()), 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': {'airflow_version': '2.1.0.dev0', 'dag_id': 'dag_pod_operatorxcom', 'execution_date': '2021-04-30T100508.1237750000-f80bbc1dc', 'kubernetes_pod_operator': 'True', 'task_id': 'bash_echo_2', 'try_number': '1'}, 'managed_fields': [{'api_version': 'v1', 'fields': None, 'manager': 'OpenAPI-Generator', 'operation': 'Update', 'time': datetime.datetime(2021, 4, 30, 10, 5, 24, tzinfo=tzlocal())}], 'name': 'bash-echo-2.c3a98f36477a43f598c786a3cb2efc55', 'namespace': 'airflow', 'owner_references': None, 'resource_version': '8416', 'self_link': None, 'uid': '4669d35e-6cfe-4863-bdc3-4fdba76ce6b4'}, 'spec': {'active_deadline_seconds': None, 'affinity': {'node_affinity': None, 'pod_affinity': None, 'pod_anti_affinity': None}, 'automount_service_account_token': None, 'containers': [{'args': ['echo', 'key1 was: value1', ',key2 was: value2', ",the entire object was: {'key1': 'value1', " "'key2': 'value2'}"], 'command': None, 'env': None, 'env_from': None, 'image': 'bash:4.4', 'image_pull_policy': 'IfNotPresent', 'lifecycle': None, 'liveness_probe': None, 'name': 'base', 'ports': None, 'readiness_probe': None, 'resources': {'limits': None, 'requests': None}, 'security_context': None, 'stdin': None, 'stdin_once': None, 'termination_message_path': '/dev/termination-log', 'termination_message_policy': 'File', 'tty': None, 'volume_devices': None, 'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount', 'mount_propagation': None, 'name': 'default-token-f546n', 'read_only': True, 'sub_path': None, 'sub_path_expr': None}], 'working_dir': None}], 'dns_config': None, 'dns_policy': 'ClusterFirst', 'enable_service_links': True, 'host_aliases': None, 'host_ipc': None, 'host_network': None, 'host_pid': None, 'hostname': None, 'image_pull_secrets': None, 'init_containers': None, 'node_name': None, 'node_selector': None, 'preemption_policy': 'PreemptLowerPriority', 'priority': 0, 'priority_class_name': None, 'readiness_gates': None, 'restart_policy': 'Never', 'runtime_class_name': None, 'scheduler_name': 'default-scheduler', 'security_context': {'fs_group': None, 'run_as_group': None, 'run_as_non_root': None, 'run_as_user': None, 'se_linux_options': None, 'supplemental_groups': None, 'sysctls': None, 'windows_options': None}, 'service_account': 'default', 'service_account_name': 'default', 'share_process_namespace': None, 'subdomain': None, 'termination_grace_period_seconds': 30, 'tolerations': [{'effect': 'NoExecute', 'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'toleration_seconds': 300, 'value': None}, {'effect': 'NoExecute', 'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'toleration_seconds': 300, 'value': None}], 'volumes': [{'aws_elastic_block_store': None, 'azure_disk': None, 'azure_file': None, 'cephfs': None, 'cinder': None, 'config_map': None, 'csi': None, 'downward_api': None, 'empty_dir': None, 'fc': None, 'flex_volume': None, 'flocker': None, 'gce_persistent_disk': None, 'git_repo': None, 'glusterfs': None, 'host_path': None, 'iscsi': None, 'name': 'default-token-f546n', 'nfs': None, 'persistent_volume_claim': None, 'photon_persistent_disk': None, 'portworx_volume': None, 'projected': None, 'quobyte': None, 'rbd': None, 'scale_io': None, 'secret': {'default_mode': 420, 'items': None, 'optional': None, 'secret_name': 'default-token-f546n'}, 'storageos': None, 'vsphere_volume': None}]}, 'status': {'conditions': None, 'container_statuses': None, 'host_ip': None, 'init_container_statuses': None, 'message': None, 'nominated_node_name': None, 'phase': 'Pending', 'pod_ip': None, 'qos_class': 'BestEffort', 'reason': None, 'start_time': None}} [2021-04-30 10:05:24,851] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8419","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"BestEffort"}} [2021-04-30 10:05:24,853] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Pending [2021-04-30 10:05:24,853] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 [2021-04-30 10:05:25,861] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8419","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"bash:4.4","imageID":"","started":false}],"qosClass":"BestEffort"}} [2021-04-30 10:05:25,863] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Pending [2021-04-30 10:05:25,863] {pod_launcher.py:126} WARNING - Pod not yet started: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 [2021-04-30 10:05:26,866] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}} [2021-04-30 10:05:26,868] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Succeeded [2021-04-30 10:05:26,868] {pod_launcher.py:302} INFO - Event with job id bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 Succeeded [2021-04-30 10:05:26,871] {pod_launcher.py:148} INFO - key1 was: value1 ,key2 was: value2 ,the entire object was: {'key1': 'value1', 'key2': 'value2'} [2021-04-30 10:05:27,881] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}} [2021-04-30 10:05:27,884] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}} [2021-04-30 10:05:27,885] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Succeeded [2021-04-30 10:05:27,885] {pod_launcher.py:302} INFO - Event with job id bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 Succeeded [2021-04-30 10:05:27,887] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"bash-echo-2.c3a98f36477a43f598c786a3cb2efc55","namespace":"airflow","uid":"4669d35e-6cfe-4863-bdc3-4fdba76ce6b4","resourceVersion":"8428","creationTimestamp":"2021-04-30T10:05:24Z","labels":{"airflow_version":"2.1.0.dev0","dag_id":"dag_pod_operatorxcom","execution_date":"2021-04-30T100508.1237750000-f80bbc1dc","kubernetes_pod_operator":"True","task_id":"bash_echo_2","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:24Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-04-30T10:05:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.47\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-f546n","secret":{"secretName":"default-token-f546n","defaultMode":420}}],"containers":[{"name":"base","image":"bash:4.4","args":["echo","key1 was: value1",",key2 was: value2",",the entire object was: {'key1': 'value1', 'key2': 'value2'}"],"resources":{},"volumeMounts":[{"name":"default-token-f546n","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"airflow-python-3.6-v1.20.2-worker","securityContext":{},"affinity":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-04-30T10:05:24Z"}],"hostIP":"172.19.0.2","podIP":"10.244.1.47","podIPs":[{"ip":"10.244.1.47"}],"startTime":"2021-04-30T10:05:24Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-04-30T10:05:25Z","finishedAt":"2021-04-30T10:05:25Z","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784"}},"lastState":{},"ready":false,"restartCount":0,"image":"docker.io/library/bash:4.4","imageID":"docker.io/library/bash@sha256:5771a6895a661cdcacf2bff9d71529dbb9eb22c09d2e05094906d5db74cec5cb","containerID":"containerd://3be4c88d6574ea28b2cc80d9392af410f91e3554f91f007f0e5da16e55c87784","started":false}],"qosClass":"BestEffort"}} [2021-04-30 10:05:27,888] {pod_launcher.py:189} INFO - Event: bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 had an event of type Succeeded [2021-04-30 10:05:27,888] {pod_launcher.py:302} INFO - Event with job id bash-echo-2.c3a98f36477a43f598c786a3cb2efc55 Succeeded [2021-04-30 10:05:27,888] {__init__.py:107} DEBUG - Lineage called with inlets: [], outlets: [] [2021-04-30 10:05:27,889] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:27,894] {taskinstance.py:630} DEBUG - Refreshed TaskInstance [2021-04-30 10:05:27,895] {taskinstance.py:1191} INFO - Marking task as SUCCESS. dag_id=dag_pod_operatorxcom, task_id=bash_echo_2, execution_date=20210430T100508, start_date=20210430T100524, end_date=20210430T100527 [2021-04-30 10:05:27,895] {taskinstance.py:1887} DEBUG - Task Duration set to 3.177347 [2021-04-30 10:05:27,912] {dagrun.py:491} DEBUG - number of tis tasks for : 0 task(s) [2021-04-30 10:05:27,913] {taskinstance.py:1245} INFO - 0 downstream tasks scheduled from follow-on schedule check [2021-04-30 10:05:27,913] {cli_action_loggers.py:84} DEBUG - Calling callbacks: [] [2021-04-30 10:05:27,918] {local_task_job.py:152} INFO - Task exited with return code 0 [2021-04-30 10:05:27,918] {taskinstance.py:595} DEBUG - Refreshing TaskInstance from DB [2021-04-30 10:05:27,926] {taskinstance.py:630} DEBUG - Refreshed TaskInstance ```
michaelosthege commented 3 years ago

@ephraimbuddy both your tasks ran with Airflow 2.1.0.? In my case the first task ran with Airflow 1.10.14 and the second with 2.0.1. They werde PythonOperators and the return value was a simple dictionary. But I can't tell if that is the same problem as with the KubernetesPodOperator.

ephraimbuddy commented 3 years ago

@michaelosthege, I tested with KubernetesPodOperator. I have not tested your case. Can you add include_prior_dates=True to the xcom_pull?

michaelosthege commented 3 years ago

@ephraimbuddy the tasks were part of the same DAG. (I cleared the downstream one after the 2.0.1 upgrade, but not the upstream task that had run on 1.10.14.) Therefore it doesn't sound like include_prior_dates should make any difference, right?

ecwootten commented 3 years ago

I wonder if this might be a wider problem than just the KubernetesPod/Python operators. I have seen this problem on 1.10.12 with a custom operator (derived from BaseOperator) - but it happens only very intermittently.

Our production cluster (AWS ECS, using docker + CeleryExecutor) has been running for ~11 months without issue, and I just saw this bug for the first time. A single xcom JSON couldn't be loaded, and that stopped all DAGs from making any progress. Deleting the relevant record from the xcom table restored everything to working order.

During development (running locally with docker + SequentialExecutor) the bug happened a few times. It seemed to be more likely to happen when the system was under heavy load, but I've not really got enough data to be sure.

eladkal commented 2 years ago

If someone experience this issue please add reproduce example. Since no comments on this issue for months + ephraim was unable to reproduce I think it's safe to close this issue.

ecwootten commented 2 years ago

Since my last comment, we've experienced this bug twice on our production cluster in 6 months of otherwise trouble-free running. So it is highly intermittent, and we have not found any way to reliably reproduce it. It is also a very problematic bug, because it completely disables the DAGs - but I appreciate that 1.10 is a pretty old version now.