No logs from pod

[barloc]

Hello. There are no logs from jobs in out environment. 'get_logs' is True. We use service account with some rights, may be we need more of it?

Logs

2022-10-06, 04:21:04 UTC] {taskinstance.py:1397} INFO - Executing <Task(KubernetesJobOperator): dbt_eds_daily> on 2022-10-05 04:21:00+00:00
[2022-10-06, 04:21:04 UTC] {standard_task_runner.py:52} INFO - Started process 28952 to run task
[2022-10-06, 04:21:04 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'dbt_daily_1', 'dbt_eds_daily', 'scheduled__2022-10-05T04:21:00+00:00', '--job-id', '260441', '--raw', '--subdir', 'DAGS_FOLDER/dags/eds/dbt_daily.py', '--cfg-path', '/tmp/tmprbn4so5r', '--error-file', '/tmp/tmpqbjlb0e3']
[2022-10-06, 04:21:04 UTC] {standard_task_runner.py:80} INFO - Job 260441: Subtask dbt_eds_daily
[2022-10-06, 04:21:04 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:528: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
  option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)

[2022-10-06, 04:21:04 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:528: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
  option = self._get_environment_variables(deprecated_key, deprecated_section, key, section)

[2022-10-06, 04:21:04 UTC] {task_command.py:371} INFO - Running <TaskInstance: dbt_daily_1.dbt_eds_daily scheduled__2022-10-05T04:21:00+00:00 [running]> on host airflow-etl-worker-1.airflow-etl-worker.dapa.svc.dapak
[2022-10-06, 04:21:04 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/utils/context.py:202: AirflowContextDeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds }}' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2022-10-06, 04:21:04 UTC] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/utils/context.py:202: AirflowContextDeprecationWarning: Accessing 'prev_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2022-10-06, 04:21:05 UTC] {taskinstance.py:1589} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=dbt_daily_1
AIRFLOW_CTX_TASK_ID=dbt_eds_daily
AIRFLOW_CTX_EXECUTION_DATE=2022-10-05T04:21:00+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-10-05T04:21:00+00:00
[2022-10-06, 04:21:05 UTC] {job_runner.py:257} INFO - {job-runner}: Executing context: dapak
[2022-10-06, 04:21:05 UTC] {job_runner.py:257} INFO - {job-runner}: Executing cluster: dapak
[2022-10-06, 04:21:05 UTC] {job_runner.py:257} INFO - {job-runner}: Started watcher for kinds: pod, service, job, deployment, configmap, secret
[2022-10-06, 04:21:05 UTC] {job_runner.py:257} INFO - {job-runner}: Watching namespaces: dapa-jobs
[2022-10-06, 04:21:05 UTC] {job_runner.py:257} INFO - {job-runner}: Waiting for dapa-jobs/dbt-eds-daily to finish...
[2022-10-06, 04:21:05 UTC] {watchers.py:330} INFO - [dapa-jobs/jobs/dbt-eds-daily] Pending
[2022-10-06, 04:21:05 UTC] {watchers.py:330} INFO - [dapa-jobs/secrets/dbt-spark-profiles] Active
[2022-10-06, 04:21:05 UTC] {operations.py:62} INFO - [dapa-jobs/secrets/dbt-spark-profiles] created
[2022-10-06, 04:21:05 UTC] {operations.py:62} INFO - [dapa-jobs/jobs/dbt-eds-daily] created
[2022-10-06, 04:21:05 UTC] {watchers.py:330} INFO - [dapa-jobs/jobs/dbt-eds-daily] Running
[2022-10-06, 04:21:16 UTC] {watchers.py:330} INFO - [dapa-jobs/jobs/dbt-eds-daily] Succeeded
[2022-10-06, 04:21:16 UTC] {job_runner.py:257} INFO - {job-runner}: Job Succeeded
[2022-10-06, 04:21:16 UTC] {job_runner.py:257} INFO - {job-runner}: Deleting resources due to policy: IfSucceeded
[2022-10-06, 04:21:16 UTC] {job_runner.py:257} INFO - {job-runner}: Deleting job..
[2022-10-06, 04:21:16 UTC] {job_runner.py:257} INFO - {job-runner}: Deleting objects: dapa-jobs/jobs/dbt-eds-daily, dapa-jobs/secrets/dbt-spark-profiles
[2022-10-06, 04:21:16 UTC] {operations.py:62} INFO - [dapa-jobs/jobs/dbt-eds-daily] deleted
[2022-10-06, 04:21:16 UTC] {operations.py:62} INFO - [dapa-jobs/secrets/dbt-spark-profiles] deleted
[2022-10-06, 04:21:16 UTC] {job_runner.py:257} INFO - {job-runner}: Job deleted
[2022-10-06, 04:21:16 UTC] {job_runner.py:257} INFO - {job-runner}: Client stopped, execution completed.
[2022-10-06, 04:21:16 UTC] {taskinstance.py:1415} INFO - Marking task as SUCCESS. dag_id=dbt_daily_1, task_id=dbt_eds_daily, execution_date=20221005T042100, start_date=20221006T042104, end_date=20221006T042116
[2022-10-06, 04:21:16 UTC] {local_task_job.py:156} INFO - Task exited with return code 0
[2022-10-06, 04:21:16 UTC] {local_task_job.py:273} INFO - 1 downstream tasks scheduled from follow-on schedule check

Versions (please complete the following information):

• Airflow 2.3.3
• K8s 1.22.8
• KubernetesJobOperator 2.0.8

Additional context K8s acls:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ airflow_role_name }}
  namespace: {{ airflow_jobs_namespace }}
rules:
  - verbs:
      - create
      - get
      - delete
      - watch
      - list
      - patch
      - update
    apiGroups:
      - batch
    resources:
      - jobs
  - verbs:
      - create
      - get
      - delete
      - watch
      - list
      - patch
      - update
    apiGroups:
      - ''
    resources:
      - configmaps
  - verbs:
      - create
      - get
      - delete
      - watch
      - list
      - patch
      - update
    apiGroups:
      - ''
    resources:
      - secrets

[barloc]

Yes, problem was in the k8s rbac. For service account needed more rights for pods:

  - verbs:
      - get
      - watch
      - list
    apiGroups:
      - ''
    resources:
      - pods/log
      - pods

May be add it to the readme?

[LamaAni]

Sure. Add a PR?

[barloc]

Hello PR: https://github.com/LamaAni/KubernetesJobOperator/pull/70