Arkto pod stuck at ContainerCreating due to cni error 'NoneType' object has no attribute 'metadata'

[Sindica]

What happened:

ubuntu@ip-172-30-0-14:~/go/src/k8s.io/arktos$ ./cluster/kubectl.sh get pods -owide -AT
TENANT   NAMESPACE     NAME                                              HASHKEY               READY   STATUS              RESTARTS   AGE     IP             NODE             NOMINATED NODE   READINESS GATES
aaa      a-ns          whoami-deployment-75db5bd457-mck2m                5865540643476425363   0/1     ContainerCreating   0          69s     <none>         ip-172-30-0-41   <none>           <none>
aaa      a-ns          whoami-deployment-75db5bd457-vcrm7                3147574529022275730   0/1     ContainerCreating   0          69s     <none>         ip-172-30-0-60   <none>           <none>

Mizar operator Log:

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

[2022-03-02 16:47:05,152] root                 [INFO    ] Unknown Error:
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 failed:
    - 1 k8sPodCreate(param=<mizar.common.wf_param.HandlerParam object at 0x7f24c15efd00>)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

[2022-03-02 16:47:05,152] root                 [INFO    ] TaskName: k8sPodCreate Name: whoami-deployment-75db5bd457-mck2m Return code is: 2 Return message is:
===== Luigi Execution Summary =====

Scheduled 1 tasks of which:
* 1 failed:
    - 1 k8sPodCreate(param=<mizar.common.wf_param.HandlerParam object at 0x7f24c15efd00>)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

[2022-03-02 16:47:05,157] luigi-interface      [INFO    ] [pid 7] Worker Worker(salt=503430966, workers=1, host=ip-172-30-0-14, username=root, pid=7) running   k8sPodCreate(param=<mizar.common.wf_param.HandlerParam object at 0x7f24c1734580>)
[2022-03-02 16:47:05,158] root                 [INFO    ] Requires k8sPodCreate
[2022-03-02 16:47:05,158] root                 [INFO    ] Run k8sPodCreate
[2022-03-02 16:47:05,159] luigi-interface      [ERROR   ] [pid 7] Worker Worker(salt=982909551, workers=1, host=ip-172-30-0-14, username=root, pid=7) failed    k8sPodCreate(param=<mizar.common.wf_param.HandlerParam object at 0x7f24c08eff70>)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.9/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/usr/local/lib/python3.9/site-packages/mizar/dp/mizar/workflows/builtins/pods/create.py", line 159, in run
    (policy_name_list, pod_label_value, namespace_label_value) = networkpolicy_util.retrieve_change_for_networkpolicy(
  File "/usr/local/lib/python3.9/site-packages/mizar/networkpolicy/networkpolicy_util.py", line 584, in retrieve_change_for_networkpolicy
    namespace_label_combination = self.get_label_combination(namespace_obj.metadata.labels)
AttributeError: 'NoneType' object has no attribute 'metadata'

[Sindica]

Not 130 blocker