kaxil
commented
2 years ago Can you try it with Airflow 2.2.3 + latest Kubernetes Provider and see if you still see the issue. We can reopen if you do
Closed Shivarp1 closed 2 years ago
kaxil
commented
2 years ago Can you try it with Airflow 2.2.3 + latest Kubernetes Provider and see if you still see the issue. We can reopen if you do
Description The task completes and XCOM returns are successful, but the volume mounts for the worker Pod is not writing to the mounted volume. The readonly flag set as "False" in the Pod request is being overwritten as "None" in the Pod response. Attached the code and debug logs below.. Are the worker pods are allowed to write to mounted NAS volumes? Thanks shivarp --- Apache Airflow version: 1.10.12 kubernetes Version: 10.0.1 OpenAPI spec version: v1.14.5 OS: Linux Red Hat 7.x
****-- KubernetesPodOperator example code
from airflow.kubernetes.volume import Volume from airflow.kubernetes.volume_mount import VolumeMount from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator ... volume_mount1 = VolumeMount('pods-volume', mount_path='/pods', sub_path=None, read_only=False) volume1 = Volume("pods-volume", {'persistentVolumeClaim':{'claimName':1234d456-podspvc'}} ) volume_mount1 = VolumeMount('dags-volume', mount_path='/dags', sub_path=None, read_only=True) volume1 = Volume("pods-volume", {'persistentVolumeClaim':{'claimName':789d123-dagspvc'}} ) task1 = KubernetesPodOperator(namespace='xyz', image='registry-localhost/container/python-3.6:latest", cmds=["bash", "-cx"],
arguments=['echo \'Hello World \' > /pods/response.txt', 'echo \'{"ret":"Hello World"} \' > /airflow/xcom/return.json' ], ...,, get_logs=True, do_xcom_push=True,security_context=pod_security_context, volumes=[volume1, volume2], volume_mounts=[volume_mount1, volume_mount2], dag=dag) ----------------------------
Task Logs: -- DEBUG - Pod Creation Request: { "apiVersion": "v1", "kind": "Pod", ... "image":"registry-localhost/container/python-3.6:latest", "volumeMount": [ { "mountPath": "/airflow/xcom", "name": "xcom" }, { "mountPath": "/dags", "name": "dags-volume" "readOnly"; true }, { "mountPath": "/pods", "name": "pods-volume" "readOnly"; false } ] volumes":[ { "name": "pods-volume" "persistentVolumeClain":{ "claimName": "123d456-podspvc" } }, { "name": "dags-volume" "persistentVolumeClain":{ "claimName": "789d123-dagspvc" } }, { "emptyDir": {}, "name": "xcom" } ] } ---------------- DEBUG - Pod Creation Response: { "apiVersion": "v1", "kind": "Pod", ... "volumeMount": [ { "mountPath": "/airflow/xcom", "name": "xcom", .... }, { "mountPath": "/dags", "name": "dags-volume" "readOnly"; True }, { "mountPath": "/pods", "name": "pods-volume" "readOnly"; None ----> **** "False" value is being replaced by the "None" value here } ] -------------------------------------**