search

stwind / airflow-on-kubernetes

Bare minimal Airflow on Kubernetes (Local, EKS, AKS)
51 stars 10 forks source link

Webserver Installation issue #7

Open anayyar82 opened 4 years ago

anayyar82 commented 4 years ago

Seems airflow installation get stuck while running below command.

I see the airflow pod is in error and shows only below 

Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  <unknown>  default-scheduler  Successfully assigned default/airflow to minikube
  Normal  Pulled     6s         kubelet, minikube  Container image "anayyar82/airflow-aks" already present on machine
  Normal  Created    6s         kubelet, minikube  Created container webserver
  Normal  Started    5s         kubelet, minikube  Started container webserver
  Normal  Pulled     5s         kubelet, minikube  Container image "anayyar82/airflow-aks" already present on machine
  Normal  Created    5s         kubelet, minikube  Created container scheduler
  Normal  Started    5s         kubelet, minikube  Started container scheduler
(base) C02YR2K0LVDQ:airflow-on-kubernetes ankur.nayyar$ kubectl get pod airflow

running the code: 


{
  "spec": {
    "containers":[{
      "name": "webserver",
      "image": "my/airflow",
      "imagePullPolicy":"IfNotPresent",
      "command": ["airflow","webserver"],
      "stdin": true,
      "tty": true,
      "env": [
        {"name":"AIRFLOW__CORE__LOAD_EXAMPLES","value":"False"},
        {"name":"AIRFLOW__CORE__SQL_ALCHEMY_CONN","value":"mysql://airflow:airflow@mysql.default/airflow"}, 
        {"name":"AIRFLOW__CORE__EXECUTOR","value":"KubernetesExecutor"},
        {"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY","value":"my/airflow"},
        {"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG","value":"latest"},
        {"name":"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST","value":"'$PWD/dags'"}
      ],
      "volumeMounts": [{"mountPath": "/var/lib/airflow/dags","name": "store"}]
    },{
      "name": "scheduler",
      "image": "my/airflow",
      "imagePullPolicy":"IfNotPresent",
      "command": ["airflow","scheduler"],
      "stdin": true,
      "tty": true,
      "env": [
        {"name":"AIRFLOW__CORE__LOAD_EXAMPLES","value":"False"},
        {"name":"AIRFLOW__CORE__SQL_ALCHEMY_CONN","value":"mysql://airflow:airflow@mysql.default/airflow"}, 
        {"name":"AIRFLOW__CORE__EXECUTOR","value":"KubernetesExecutor"},
        {"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY","value":"my/airflow"},
        {"name":"AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG","value":"latest"},
        {"name":"AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST","value":"'$PWD/dags'"},
        {"name":"AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS","value":""}
      ],
      "volumeMounts": [{"mountPath": "/var/lib/airflow/dags","name": "store"}]
    }],
    "volumes": [{"name":"store","hostPath":{"path":"'$PWD/dags'","type":"Directory"}}]
  }
}'```
stwind commented 4 years ago

Hi @anayyar82 , you can use kubectl logs to see detail logs inside the container.

anayyar82 commented 4 years ago

Thanks @stwind Yeah i did but seems it stucked somehow.

kubectl logs -f airflow -c scheduler 

[2020-04-06 14:33:31,551] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-04-06 14:33:33,244] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2020-04-06 14:33:33,303] {scheduler_job.py:1344} INFO - Starting the scheduler
[2020-04-06 14:33:33,303] {scheduler_job.py:1352} INFO - Running execute loop for -1 seconds
[2020-04-06 14:33:33,305] {scheduler_job.py:1353} INFO - Processing each file at most -1 times
[2020-04-06 14:33:33,306] {scheduler_job.py:1356} INFO - Searching for files in /tmp/airflow/dags
[2020-04-06 14:33:33,308] {scheduler_job.py:1358} INFO - There are 0 files in /tmp/airflow/dags
[2020-04-06 14:33:33,317] {kubernetes_executor.py:805} INFO - Start Kubernetes executor
[2020-04-06 14:33:53,879] {kubernetes_executor.py:351} INFO - Event: and now my watch begins starting at resource_version: 0
[2020-04-06 14:33:36,929] {kubernetes_executor.py:733} INFO - When executor started up, found 0 queued task instances
[2020-04-06 14:33:53,933] {scheduler_job.py:1409} INFO - Resetting orphaned tasks for active dag runs
[2020-04-06 14:33:54,212] {dag_processing.py:556} INFO - Launched DagFileProcessorManager with pid: 29
[2020-04-06 14:33:54,664] {settings.py:54} INFO - Configured default timezone <Timezone [UTC]>
[2020-04-06 14:33:54,693] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=29

and for Webserver

kubectl logs -f airflow -c webserver
[2020-04-06 14:33:31,551] {settings.py:253} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
anayyar82 commented 4 years ago

May be this helps 

 kubectl describe  pod airflow
Name:         airflow
Namespace:    default
Priority:     0
Node:         minikube/192.168.64.2
Start Time:   Mon, 06 Apr 2020 09:33:11 -0500
Labels:       run=airflow
Annotations:  <none>
Status:       Running
IP:           172.17.0.2
IPs:
  IP:  172.17.0.2
Containers:
  webserver:
    Container ID:  docker://f09249e339079acd78e2175ef690a853dd264664bf53ac826f743c46e83bc7b4
    Image:         anayyar82/airflow-aks1
    Image ID:      docker-pullable://anayyar82/airflow-aks1@sha256:7f459e5404ab93668ed3a9476ac02cda8acc691aae1b8b983e65611109c546be
    Port:          <none>
    Host Port:     <none>
    Command:
      airflow
      webserver
    State:          Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 06 Apr 2020 09:33:30 -0500
      Finished:     Mon, 06 Apr 2020 09:33:53 -0500
    Ready:          False
    Restart Count:  0
    Environment:
      AIRFLOW__CORE__LOAD_EXAMPLES:                      False
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:                   mysql://airflow:airflow@mysql.default/airflow
      AIRFLOW__CORE__EXECUTOR:                           KubernetesExecutor
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY:  anayyar82/airflow-aks1
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG:         latest
      AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST:             /tmp/airflow/dags
    Mounts:
      /tmp/airflow/dags from store (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-n8qt9 (ro)
  scheduler:
    Container ID:  docker://8a6731a2e07caa9e0f58cbc26a94e7c897fc3354e2d8e593461db946c8fba763
    Image:         anayyar82/airflow-aks1
    Image ID:      docker-pullable://anayyar82/airflow-aks1@sha256:7f459e5404ab93668ed3a9476ac02cda8acc691aae1b8b983e65611109c546be
    Port:          <none>
    Host Port:     <none>
    Command:
      airflow
      scheduler
    State:          Running
      Started:      Mon, 06 Apr 2020 09:33:30 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      AIRFLOW__CORE__LOAD_EXAMPLES:                      False
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:                   mysql://airflow:airflow@mysql.default/airflow
      AIRFLOW__CORE__EXECUTOR:                           KubernetesExecutor
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY:  anayyar82/airflow-aks1
      AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG:         latest
      AIRFLOW__KUBERNETES__DAGS_VOLUME_HOST:             /tmp/airflow/dags
      AIRFLOW__KUBERNETES__KUBE_CLIENT_REQUEST_ARGS:
    Mounts:
      /tmp/airflow/dags from store (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-n8qt9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  store:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/airflow/dags
    HostPathType:
  default-token-n8qt9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-n8qt9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
stwind commented 4 years ago

sorry by just looking at the logs I am not quite sure what is the problem, maybe you could try running without k8s first?

anayyar82 commented 4 years ago

Yeah without it's running fine. Could you please recheck if something is missing in the doc?