ex-pod-affinity sample failed after follow the guide

[noppGithub]

In which file did you encounter the issue?

file: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/HEAD/composer/workflows/kubernetes_pod_operator.py

Did you change the file? If so, how?

No

Describe the issue

My Composer info: Zone: asia-southeast1 Composer version: 1.19.0 Airflow version: 2.2.5

I follow the guide https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator on the section of Pod Affinity Configuration, I created a node-pool name pool-0, the node has been created with return message below.

NAME    MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
pool-0  n1-standard-1  100           1.21.11-gke.1900

I back to the Composer and run again(as the guide suggested) but got the error below in the Composer log.

*** Reading remote log from gs://asia-southeast1-cc1-2e842d2e-bucket/logs/composer_sample_kubernetes_pod/ex-pod-affinity/2022-06-24T08:20:54.389986+00:00/12.log.
[2022-06-26, 08:07:05 +07] {taskinstance.py:1044} INFO - Dependencies all met for <TaskInstance: composer_sample_kubernetes_pod.ex-pod-affinity scheduled__2022-06-24T08:20:54.389986+00:00 [queued]>
[2022-06-26, 08:07:05 +07] {taskinstance.py:1044} INFO - Dependencies all met for <TaskInstance: composer_sample_kubernetes_pod.ex-pod-affinity scheduled__2022-06-24T08:20:54.389986+00:00 [queued]>
[2022-06-26, 08:07:05 +07] {taskinstance.py:1250} INFO - 
--------------------------------------------------------------------------------
[2022-06-26, 08:07:05 +07] {taskinstance.py:1251} INFO - Starting attempt 12 of 14
[2022-06-26, 08:07:05 +07] {taskinstance.py:1252} INFO - 
--------------------------------------------------------------------------------
[2022-06-26, 08:07:05 +07] {taskinstance.py:1271} INFO - Executing <Task(KubernetesPodOperator): ex-pod-affinity> on 2022-06-24 08:20:54.389986+00:00
[2022-06-26, 08:07:05 +07] {standard_task_runner.py:52} INFO - Started process 47413 to run task
[2022-06-26, 08:07:05 +07] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'composer_sample_kubernetes_pod', 'ex-pod-affinity', 'scheduled__2022-06-24T08:20:54.389986+00:00', '--job-id', '700', '--raw', '--subdir', 'DAGS_FOLDER/kubernetes_pod_operator.py', '--cfg-path', '/tmp/tmpkgdujdr7', '--error-file', '/tmp/tmp8bz_8iwy']
[2022-06-26, 08:07:05 +07] {standard_task_runner.py:80} INFO - Job 700: Subtask ex-pod-affinity
[2022-06-26, 08:07:06 +07] {task_command.py:298} INFO - Running <TaskInstance: composer_sample_kubernetes_pod.ex-pod-affinity scheduled__2022-06-24T08:20:54.389986+00:00 [running]> on host airflow-worker-8696b4747f-n59j4
[2022-06-26, 08:07:06 +07] {taskinstance.py:1448} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=composer_sample_kubernetes_pod
AIRFLOW_CTX_TASK_ID=ex-pod-affinity
AIRFLOW_CTX_EXECUTION_DATE=2022-06-24T08:20:54.389986+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-06-24T08:20:54.389986+00:00
[2022-06-26, 08:07:06 +07] {kubernetes_pod.py:566} INFO - Creating pod ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69 with labels: {'dag_id': 'composer_sample_kubernetes_pod', 'task_id': 'ex-pod-affinity', 'execution_date': '2022-06-24T082054.3899860000-0c9c100e3', 'try_number': '12'}
[2022-06-26, 08:07:07 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:08 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:09 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:10 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:11 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:12 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:13 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:14 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:15 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:16 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:17 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:18 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:19 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:20 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:21 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:22 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:23 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:24 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:25 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:26 +07] {pod_manager.py:157} WARNING - Pod not yet started: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:32 +07] {pod_manager.py:197} INFO - Can't use an undefined value as an ARRAY reference at /usr/local/lib/perl5/5.36.0/Math/BigInt/Calc.pm line 1049.
[2022-06-26, 08:07:35 +07] {kubernetes_pod.py:459} INFO - Deleting pod: ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69
[2022-06-26, 08:07:35 +07] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 430, in execute
    self.cleanup(
  File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 452, in cleanup
    raise AirflowException(f'Pod {pod and pod.metadata.name} returned a failure: {remote_pod}')
airflow.exceptions.AirflowException: Pod ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69 returned a failure: {'api_version': 'v1',
 'kind': 'Pod',
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': {'airflow_version': '2.2.5-composer',
                         'dag_id': 'composer_sample_kubernetes_pod',
                         'execution_date': '2022-06-24T082054.3899860000-0c9c100e3',
                         'kubernetes_pod_operator': 'True',
                         'task_id': 'ex-pod-affinity',
                         'try_number': '12'},
              'managed_fields': [{'api_version': 'v1',
                                  'fields': None,
                                  'manager': 'OpenAPI-Generator',
                                  'operation': 'Update',
                                  'time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())},
                                 {'api_version': 'v1',
                                  'fields': None,
                                  'manager': 'kubelet',
                                  'operation': 'Update',
                                  'time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal())}],
              'name': 'ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69',
              'namespace': 'default',
              'owner_references': None,
              'resource_version': '981544',
              'self_link': None,
              'uid': '80edd551-f55b-46d1-972c-bff97473d074'},
 'spec': {'active_deadline_seconds': None,
          'affinity': {'node_affinity': {'preferred_during_scheduling_ignored_during_execution': None,
                                         'required_during_scheduling_ignored_during_execution': {'node_selector_terms': [{'match_expressions': [{'key': 'cloud.google.com/gke-nodepool',
                                                                                                                                                 'operator': 'In',
                                                                                                                                                 'values': ['pool-0',
                                                                                                                                                            'pool-1']}],
                                                                                                                          'match_fields': None}]}},
                       'pod_affinity': None,
                       'pod_anti_affinity': None},
          'automount_service_account_token': None,
          'containers': [{'args': ['-Mbignum=bpi', '-wle', 'print bpi(2000)'],
                          'command': ['perl'],
                          'env': None,
                          'env_from': None,
                          'image': 'perl',
                          'image_pull_policy': 'Always',
                          'lifecycle': None,
                          'liveness_probe': None,
                          'name': 'base',
                          'ports': None,
                          'readiness_probe': None,
                          'resources': {'limits': None, 'requests': None},
                          'security_context': None,
                          'stdin': None,
                          'stdin_once': None,
                          'termination_message_path': '/dev/termination-log',
                          'termination_message_policy': 'File',
                          'tty': None,
                          'volume_devices': None,
                          'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
                                             'mount_propagation': None,
                                             'name': 'kube-api-access-lw9pt',
                                             'read_only': True,
                                             'sub_path': None,
                                             'sub_path_expr': None}],
                          'working_dir': None}],
          'dns_config': None,
          'dns_policy': 'ClusterFirst',
          'enable_service_links': True,
          'host_aliases': None,
          'host_ipc': None,
          'host_network': None,
          'host_pid': None,
          'hostname': None,
          'image_pull_secrets': None,
          'init_containers': None,
          'node_name': 'gke-asia-southeast1-cc1-2e842d-pool-0-32a57203-s34t',
          'node_selector': None,
          'preemption_policy': 'PreemptLowerPriority',
          'priority': 0,
          'priority_class_name': None,
          'readiness_gates': None,
          'restart_policy': 'Never',
          'runtime_class_name': None,
          'scheduler_name': 'default-scheduler',
          'security_context': {'fs_group': None,
                               'run_as_group': None,
                               'run_as_non_root': None,
                               'run_as_user': None,
                               'se_linux_options': None,
                               'supplemental_groups': None,
                               'sysctls': None,
                               'windows_options': None},
          'service_account': 'default',
          'service_account_name': 'default',
          'share_process_namespace': None,
          'subdomain': None,
          'termination_grace_period_seconds': 30,
          'tolerations': [{'effect': 'NoExecute',
                           'key': 'node.kubernetes.io/not-ready',
                           'operator': 'Exists',
                           'toleration_seconds': 300,
                           'value': None},
                          {'effect': 'NoExecute',
                           'key': 'node.kubernetes.io/unreachable',
                           'operator': 'Exists',
                           'toleration_seconds': 300,
                           'value': None}],
          'volumes': [{'aws_elastic_block_store': None,
                       'azure_disk': None,
                       'azure_file': None,
                       'cephfs': None,
                       'cinder': None,
                       'config_map': None,
                       'csi': None,
                       'downward_api': None,
                       'empty_dir': None,
                       'fc': None,
                       'flex_volume': None,
                       'flocker': None,
                       'gce_persistent_disk': None,
                       'git_repo': None,
                       'glusterfs': None,
                       'host_path': None,
                       'iscsi': None,
                       'name': 'kube-api-access-lw9pt',
                       'nfs': None,
                       'persistent_volume_claim': None,
                       'photon_persistent_disk': None,
                       'portworx_volume': None,
                       'projected': {'default_mode': 420,
                                     'sources': [{'config_map': None,
                                                  'downward_api': None,
                                                  'secret': None,
                                                  'service_account_token': {'audience': None,
                                                                            'expiration_seconds': 3607,
                                                                            'path': 'token'}},
                                                 {'config_map': {'items': [{'key': 'ca.crt',
                                                                            'mode': None,
                                                                            'path': 'ca.crt'}],
                                                                 'name': 'kube-root-ca.crt',
                                                                 'optional': None},
                                                  'downward_api': None,
                                                  'secret': None,
                                                  'service_account_token': None},
                                                 {'config_map': None,
                                                  'downward_api': {'items': [{'field_ref': {'api_version': 'v1',
                                                                                            'field_path': 'metadata.namespace'},
                                                                              'mode': None,
                                                                              'path': 'namespace',
                                                                              'resource_field_ref': None}]},
                                                  'secret': None,
                                                  'service_account_token': None}]},
                       'quobyte': None,
                       'rbd': None,
                       'scale_io': None,
                       'secret': None,
                       'storageos': None,
                       'vsphere_volume': None}]},
 'status': {'conditions': [{'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
                            'message': None,
                            'reason': None,
                            'status': 'True',
                            'type': 'Initialized'},
                           {'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
                            'message': 'containers with unready status: [base]',
                            'reason': 'ContainersNotReady',
                            'status': 'False',
                            'type': 'Ready'},
                           {'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
                            'message': 'containers with unready status: [base]',
                            'reason': 'ContainersNotReady',
                            'status': 'False',
                            'type': 'ContainersReady'},
                           {'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
                            'message': None,
                            'reason': None,
                            'status': 'True',
                            'type': 'PodScheduled'}],
            'container_statuses': [{'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
                                    'image': 'docker.io/library/perl:latest',
                                    'image_id': 'docker.io/library/perl@sha256:b83504159488e1ae88e6705f34b78fac623b2cd55ca0304de31bfb09d59f7bfd',
                                    'last_state': {'running': None,
                                                   'terminated': None,
                                                   'waiting': None},
                                    'name': 'base',
                                    'ready': False,
                                    'restart_count': 0,
                                    'state': {'running': None,
                                              'terminated': {'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
                                                             'exit_code': 255,
                                                             'finished_at': datetime.datetime(2022, 6, 26, 1, 7, 32, tzinfo=tzlocal()),
                                                             'message': None,
                                                             'reason': 'Error',
                                                             'signal': None,
                                                             'started_at': datetime.datetime(2022, 6, 26, 1, 7, 26, tzinfo=tzlocal())},
                                              'waiting': None}}],
            'host_ip': '10.148.0.17',
            'init_container_statuses': None,
            'message': None,
            'nominated_node_name': None,
            'phase': 'Failed',
            'pod_ip': '10.124.3.2',
            'qos_class': 'BestEffort',
            'reason': None,
            'start_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())}}
[2022-06-26, 08:07:35 +07] {taskinstance.py:1279} INFO - Marking task as UP_FOR_RETRY. dag_id=composer_sample_kubernetes_pod, task_id=ex-pod-affinity, execution_date=20220624T082054, start_date=20220626T010705, end_date=20220626T010735
[2022-06-26, 08:07:35 +07] {standard_task_runner.py:93} ERROR - Failed to execute job 700 for task ex-pod-affinity (Pod ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69 returned a failure: {'api_version': 'v1',
 'kind': 'Pod',
 'metadata': {'annotations': None,
              'cluster_name': None,
              'creation_timestamp': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
              'deletion_grace_period_seconds': None,
              'deletion_timestamp': None,
              'finalizers': None,
              'generate_name': None,
              'generation': None,
              'initializers': None,
              'labels': {'airflow_version': '2.2.5-composer',
                         'dag_id': 'composer_sample_kubernetes_pod',
                         'execution_date': '2022-06-24T082054.3899860000-0c9c100e3',
                         'kubernetes_pod_operator': 'True',
                         'task_id': 'ex-pod-affinity',
                         'try_number': '12'},
              'managed_fields': [{'api_version': 'v1',
                                  'fields': None,
                                  'manager': 'OpenAPI-Generator',
                                  'operation': 'Update',
                                  'time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())},
                                 {'api_version': 'v1',
                                  'fields': None,
                                  'manager': 'kubelet',
                                  'operation': 'Update',
                                  'time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal())}],
              'name': 'ex-pod-affinity.4188bcd51e2543d9a91c95828c870f69',
              'namespace': 'default',
              'owner_references': None,
              'resource_version': '981544',
              'self_link': None,
              'uid': '80edd551-f55b-46d1-972c-bff97473d074'},
 'spec': {'active_deadline_seconds': None,
          'affinity': {'node_affinity': {'preferred_during_scheduling_ignored_during_execution': None,
                                         'required_during_scheduling_ignored_during_execution': {'node_selector_terms': [{'match_expressions': [{'key': 'cloud.google.com/gke-nodepool',
                                                                                                                                                 'operator': 'In',
                                                                                                                                                 'values': ['pool-0',
                                                                                                                                                            'pool-1']}],
                                                                                                                          'match_fields': None}]}},
                       'pod_affinity': None,
                       'pod_anti_affinity': None},
          'automount_service_account_token': None,
          'containers': [{'args': ['-Mbignum=bpi', '-wle', 'print bpi(2000)'],
                          'command': ['perl'],
                          'env': None,
                          'env_from': None,
                          'image': 'perl',
                          'image_pull_policy': 'Always',
                          'lifecycle': None,
                          'liveness_probe': None,
                          'name': 'base',
                          'ports': None,
                          'readiness_probe': None,
                          'resources': {'limits': None, 'requests': None},
                          'security_context': None,
                          'stdin': None,
                          'stdin_once': None,
                          'termination_message_path': '/dev/termination-log',
                          'termination_message_policy': 'File',
                          'tty': None,
                          'volume_devices': None,
                          'volume_mounts': [{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
                                             'mount_propagation': None,
                                             'name': 'kube-api-access-lw9pt',
                                             'read_only': True,
                                             'sub_path': None,
                                             'sub_path_expr': None}],
                          'working_dir': None}],
          'dns_config': None,
          'dns_policy': 'ClusterFirst',
          'enable_service_links': True,
          'host_aliases': None,
          'host_ipc': None,
          'host_network': None,
          'host_pid': None,
          'hostname': None,
          'image_pull_secrets': None,
          'init_containers': None,
          'node_name': 'gke-asia-southeast1-cc1-2e842d-pool-0-32a57203-s34t',
          'node_selector': None,
          'preemption_policy': 'PreemptLowerPriority',
          'priority': 0,
          'priority_class_name': None,
          'readiness_gates': None,
          'restart_policy': 'Never',
          'runtime_class_name': None,
          'scheduler_name': 'default-scheduler',
          'security_context': {'fs_group': None,
                               'run_as_group': None,
                               'run_as_non_root': None,
                               'run_as_user': None,
                               'se_linux_options': None,
                               'supplemental_groups': None,
                               'sysctls': None,
                               'windows_options': None},
          'service_account': 'default',
          'service_account_name': 'default',
          'share_process_namespace': None,
          'subdomain': None,
          'termination_grace_period_seconds': 30,
          'tolerations': [{'effect': 'NoExecute',
                           'key': 'node.kubernetes.io/not-ready',
                           'operator': 'Exists',
                           'toleration_seconds': 300,
                           'value': None},
                          {'effect': 'NoExecute',
                           'key': 'node.kubernetes.io/unreachable',
                           'operator': 'Exists',
                           'toleration_seconds': 300,
                           'value': None}],
          'volumes': [{'aws_elastic_block_store': None,
                       'azure_disk': None,
                       'azure_file': None,
                       'cephfs': None,
                       'cinder': None,
                       'config_map': None,
                       'csi': None,
                       'downward_api': None,
                       'empty_dir': None,
                       'fc': None,
                       'flex_volume': None,
                       'flocker': None,
                       'gce_persistent_disk': None,
                       'git_repo': None,
                       'glusterfs': None,
                       'host_path': None,
                       'iscsi': None,
                       'name': 'kube-api-access-lw9pt',
                       'nfs': None,
                       'persistent_volume_claim': None,
                       'photon_persistent_disk': None,
                       'portworx_volume': None,
                       'projected': {'default_mode': 420,
                                     'sources': [{'config_map': None,
                                                  'downward_api': None,
                                                  'secret': None,
                                                  'service_account_token': {'audience': None,
                                                                            'expiration_seconds': 3607,
                                                                            'path': 'token'}},
                                                 {'config_map': {'items': [{'key': 'ca.crt',
                                                                            'mode': None,
                                                                            'path': 'ca.crt'}],
                                                                 'name': 'kube-root-ca.crt',
                                                                 'optional': None},
                                                  'downward_api': None,
                                                  'secret': None,
                                                  'service_account_token': None},
                                                 {'config_map': None,
                                                  'downward_api': {'items': [{'field_ref': {'api_version': 'v1',
                                                                                            'field_path': 'metadata.namespace'},
                                                                              'mode': None,
                                                                              'path': 'namespace',
                                                                              'resource_field_ref': None}]},
                                                  'secret': None,
                                                  'service_account_token': None}]},
                       'quobyte': None,
                       'rbd': None,
                       'scale_io': None,
                       'secret': None,
                       'storageos': None,
                       'vsphere_volume': None}]},
 'status': {'conditions': [{'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
                            'message': None,
                            'reason': None,
                            'status': 'True',
                            'type': 'Initialized'},
                           {'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
                            'message': 'containers with unready status: [base]',
                            'reason': 'ContainersNotReady',
                            'status': 'False',
                            'type': 'Ready'},
                           {'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 34, tzinfo=tzlocal()),
                            'message': 'containers with unready status: [base]',
                            'reason': 'ContainersNotReady',
                            'status': 'False',
                            'type': 'ContainersReady'},
                           {'last_probe_time': None,
                            'last_transition_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal()),
                            'message': None,
                            'reason': None,
                            'status': 'True',
                            'type': 'PodScheduled'}],
            'container_statuses': [{'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
                                    'image': 'docker.io/library/perl:latest',
                                    'image_id': 'docker.io/library/perl@sha256:b83504159488e1ae88e6705f34b78fac623b2cd55ca0304de31bfb09d59f7bfd',
                                    'last_state': {'running': None,
                                                   'terminated': None,
                                                   'waiting': None},
                                    'name': 'base',
                                    'ready': False,
                                    'restart_count': 0,
                                    'state': {'running': None,
                                              'terminated': {'container_id': 'containerd://265b273c470d3e0243913d01440d8d28353d5036916a39809ce03fb3d49d4c4f',
                                                             'exit_code': 255,
                                                             'finished_at': datetime.datetime(2022, 6, 26, 1, 7, 32, tzinfo=tzlocal()),
                                                             'message': None,
                                                             'reason': 'Error',
                                                             'signal': None,
                                                             'started_at': datetime.datetime(2022, 6, 26, 1, 7, 26, tzinfo=tzlocal())},
                                              'waiting': None}}],
            'host_ip': '10.148.0.17',
            'init_container_statuses': None,
            'message': None,
            'nominated_node_name': None,
            'phase': 'Failed',
            'pod_ip': '10.124.3.2',
            'qos_class': 'BestEffort',
            'reason': None,
            'start_time': datetime.datetime(2022, 6, 26, 1, 7, 7, tzinfo=tzlocal())}}; 47413)

[leahecole]

Hi - thanks for posting this issue @noppGithub! If you do have a support contract with Google, do please also open an issue there. If not, I'll try to take a look at this at some point.

What networking options do you have for your environment? (Is it public IP? Private IP? using VPC-SC? Any related details can help me there)

[noppGithub]

@leahecole Thank you so much.

I don't have exclusive support with Google, this is just a normal test account to make sure that I can use this GCP service properly. Please let me know the right way to request support from Google on this issue.

For the networking options, the guide provided by Google, there is no network setting during the setup all of the services below are connected to the Google network.

• Composer
• GKE
• GCS
• Pub/Sub

Please let me know if you need access to the GCP account that I run this test, I can give you access so you can check for the required information.

[leahecole]

Thanks for the clarifying network! I'll do my best to replicate the issue and get back to you.

[leahecole]

I was able to replicate this! Now, to figure out a solution 😁

[noppGithub]

I was able to replicate this! Now, to figure out a solution 😁

Thank you so much, if anything I can do please let me know.

[noppGithub]

@leahecole Hello, is there any update about this sir.

[mhenc]

This is similar issue to https://github.com/kubernetes/website/issues/34073 fixed in https://github.com/kubernetes/website/pull/34414/files please pin the image version to perl:5.34.0