search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.39k stars 14.11k forks source link

Attempt to use Pod Template File with KubernetesExecutor version 1.10.13 Throws AttributeError: 'V1EnvFromSource' object has no attribute 'name' #12960

Closed hedrickw closed 3 years ago

hedrickw commented 3 years ago

Apache Airflow version: 1.10.13

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.18.8" Environment: Ubuntu 16.04

What happened: Attempt to use Pod Template File with KubernetesExecutor version 1.10.13 Throws AttributeError: 'V1EnvFromSource' object has no attribute 'name'

Template file being sent to pod


kind: Pod
apiVersion: v1
metadata:
name: mando_yaml
namespace: default
labels:
mylabel: mando_pod
spec:
containers:
- name: mando
image: example
env:
- name: AIRFLOW__CORE_DAGS_FOLDER
value: /src/example/apache
- name: AIRFLOW_HOME
value: /src/example/apache
- name: AIRFLOW__CORE__EXECUTOR
value: LocalExecutor
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
value: postgresql+psycopg2://postgres:airflow@example-postgresql:5432/airflow
envFrom:
- configMapRef:
name: example-env
imagePullPolicy: Always
ports: []
volumeMounts:
- mountPath: /opt/airflow/logs
name: airflow-logs
resources:
limits:
memory: "500Mi"
requests:
memory: "500Mi"
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
hostNetwork: false
imagePullSecrets: []
initContainers: []
nodeSelector: {}
restartPolicy: Never
securityContext:
runAsUser: 0
serviceAccountName: example
volumes:
- emptyDir: {}
name: airflow-logs

> Error

['airflow', 'run', 'mandalore_pod_template_limits_v2', 'mandalore_saying', '2020-12-09T16:07:19.537177+00:00', '--local', '--pool', 'default_pool', '-sd', '/src/examples/apache/dags/mando_pod_template_v2.py'], {'api_version': 'v1', 'kind': 'Pod', 'metadata': {'annotations': None, 'cluster_name': None, 'creation_timestamp': None, 'deletion_grace_period_seconds': None, 'deletion_timestamp': None, 'finalizers': None, 'generate_name': None, 'generation': None, 'initializers': None, 'labels': {'mylabel': 'mando_pod'}, 'managed_fields': None, 'name': 'mando_yaml-a2520ce7604c40b2a84338cd639b8815', 'namespace': 'default', 'owner_references': None, 'resource_version': None, 'self_link': None, 'uid': None}, 'spec': {'active_deadline_seconds': None, 'affinity': None, 'automount_service_account_token': None, 'containers': [{'args': None, 'command': None, 'env': [{'name': 'AIRFLOWCORE_DAGS_FOLDER', 'value': '/src/example/apache', 'value_from': None}, {'name': 'AIRFLOW_HOME', 'value': '/src/example/apache', 'value_from': None}, {'name': 'AIRFLOWCOREEXECUTOR', 'value': 'LocalExecutor', 'value_from': None}, {'name': 'AIRFLOWCORE__SQL_ALCHEMY_CONN', 'value': 'postgresql+psycopg2://postgres:airflow@example-postgresql:5432/airflow', 'value_from': NonE 'env_from': [{'config_map_ref': {'name': 'example-env', 'optional': None}, 'prefix': None, 'secret_ref': None}], 'image': 'example', 'image_pull_policy': 'Always', 'lifecycle': None, 'liveness_probe': None, 'name': 'mando', 'ports': [], 'readiness_probe': None, 'resources': {'limits': {'memory': '500Mi'}, 'requests': {'memory': '500Mi'}}, 'security_context': None, 'stdin': None, 'stdin_once': None, 'termination_message_path': '/dev/termination-log', 'termination_message_policy': 'File', 'tty': None, 'volume_devices': None, 'volume_mounts': [{'mount_path': '/opt/airflow/logs', 'mount_propagation': None, 'name': 'airflow-logs', 'read_only': None, 'sub_path': None, 'sub_path_expr': None}], 'working_dir': None}], 'dns_config': None, 'dns_policy': None, 'enable_service_links': None, 'host_aliases': None, 'host_ipc': None, 'host_network': False, 'host_pid': None, 'hostname': None, 'image_pull_secrets': [], 'init_containers': [], 'node_name': None, 'node_selector': {}, 'preemption_policy': None, 'priority': None, 'priority_class_name': None, 'readiness_gates': None, 'restart_policy': 'Never', 'runtime_class_name': None, 'scheduler_name': None, 'security_context': {'fs_group': None, 'run_as_group': None, 'run_as_non_root': None, 'run_as_user': 0, 'se_linux_options': None, 'supplemental_groups': None, 'sysctls': None, 'windows_options': None}, 'service_account': None, 'service_account_name': 'odin', 'share_process_namespace': None, 'subdomain': None, 'termination_grace_period_seconds': None, 'tolerations': None, 'volumes': [{'aws_elastic_block_store': None, 'azure_disk': None, 'azure_file': None, 'cephfs': None, 'cinder': None, 'config_map': None, 'csi': None, 'downward_api': None, 'empty_dir': {'medium': None, 'size_limit': None}, 'fc': None, 'flex_volume': None, 'flocker': None, 'gce_persistent_disk': None, 'git_repo': None, 'glusterfs': None, 'host_path': None, 'iscsi': None, 'name': 'airflow-logs', 'nfs': None, 'persistent_volume_claim': None, 'photon_persistent_disk': None, 'portworx_volume': None, 'projected': None, 'quobyte': None, 'rbd': None, 'scale_io': None, 'secret': None, 'storageos': None, 'vsphere_volume': None}]}, 'status': None}) [2020-12-09 16:43:20,697] {scheduler_job.py:1384} ERROR - Exception when executing execute_helper Traceback (most recent call last): File "/opt/app-root/lib64/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1382, in _execute self._execute_helper() File "/opt/app-root/lib64/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1453, in _execute_helper if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag): File "/opt/app-root/lib64/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1515, in _validate_and_run_task_instances self.executor.heartbeat() File "/opt/app-root/lib64/python3.8/site-packages/airflow/executors/base_executor.py", line 134, in heartbeat self.sync() File "/opt/app-root/lib64/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 832, in sync self.kube_scheduler.run_next(task) File "/opt/app-root/lib64/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 436, in run_next pod = PodGenerator.construct_pod( File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 564, in construct_pod return reduce(PodGenerator.reconcile_pods, pod_list) File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 429, in reconcile_pods client_pod_cp.spec = PodGenerator.reconcile_specs(base_pod.spec, client_pod_cp.spec) File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 475, in reconcile_specs client_spec.containers = PodGenerator.reconcile_containers( File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 509, in reconcile_containers client_container = extend_object_field(base_container, client_container, 'env_from') File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 647, in extend_object_field base_obj_set = _get_dict_from_list(base_obj_field, field_to_merge) File "/opt/app-root/lib64/python3.8/site-packages/airflow/kubernetes/pod_generator.py", line 676, in _get_dict_from_list result[getattr(obj, field_to_merge)] = obj AttributeError: 'V1EnvFromSource' object has no attribute 'name'

from datetime import timedelta from airflow import DAG from airflow.operators.python_operator import PythonOperator from airflow.utils.dates import days_ago from kubernetes.client import models as k8s import os

def mando_saying(): print("This is the Way") print(os.getenv("TEST"))

default_args = { "owner": "airflow", "depends_on_past": False, "start_date": days_ago(2), "email": ["airflow@example.com"], "email_on_failure": False, "email_on_retry": False, "retries": 0, "retry_delay": timedelta(minutes=5), }

dag = DAG("mandalore_pod_template_limits_v2", default_args=default_args, description="Test Script", schedule_interval=None)

this_is_the_way = PythonOperator( task_id="mandalore_saying", python_callable=mando_saying, dag=dag, depends_on_past=False, executor_config={"KubernetesExecutor": {"pod_template_file": "/src/explorer2/apache/pod_templates/mando.yaml",}}, )



I tried doing it this way because when i change the executor_config to 

`executor_config={"pod_template_file": "/src/explorer2/apache/pod_templates/mando.yaml",},`

The Scheduler is not reading the change and just launches the pod with all the defaults :/
boring-cyborg[bot] commented 3 years ago

Thanks for opening your first issue here! Be sure to follow the issue template!