[Failed to deploy] kubernetes.utils.create_from_yaml.FailToCreateError

[caohch-1]

I start a cluster with 5 nodes and deploy prometheus on it as follow:

xxx@cxxx:~/Desktop$ minikube start --nodes 5 --cpus 4 --memory 8192 --kubernetes-version=v.123.8

xxx@cxxx:~/Desktop$ kubectl get nodes
NAME           STATUS   ROLES                  AGE     VERSION
minikube       Ready    control-plane,master   3h43m   v1.23.8
minikube-m02   Ready    <none>                 3h42m   v1.23.8
minikube-m03   Ready    <none>                 3h42m   v1.23.8
minikube-m04   Ready    <none>                 3h42m   v1.23.8
minikube-m05   Ready    <none>                 3h42m   v1.23.8

xxx@cxxx:~/Desktop$ kubectl get pods -n kube-prometheus
NAME                                                 READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-0                            1/1     Running   0          137m
prometheus-kube-state-metrics-5f7c877c9f-9mf8f       1/1     Running   0          100m
prometheus-prometheus-node-exporter-5z595            1/1     Running   0          137m
prometheus-prometheus-node-exporter-79lkr            1/1     Running   0          137m
prometheus-prometheus-node-exporter-c66jn            1/1     Running   0          137m
prometheus-prometheus-node-exporter-l2zmk            1/1     Running   0          137m
prometheus-prometheus-node-exporter-wblkh            1/1     Running   0          137m
prometheus-prometheus-pushgateway-6445c657fc-xll47   1/1     Running   0          137m
prometheus-server-6db85cd89-djbzf                    2/2     Running   0          99m

I install the python dependencies in requirements.txt and change Erms-main/configs/hotel-global/yaml as follows and I also modify the CONFIG variable in AE/scripts/utils:

data_path: data_hotel
figure_path: data_hotel/figures
yaml_repo_path: yamlRepository/hotelReservation
namespace: hotel-reserv
app_img: nicklin9907/erms:hotelreservation-1.0
nodes_for_test:
- minikube-m02
- minikube-m03
- minikube-m04

prometheus_host: http://localhost:30090
nodes_for_infra:
- minikube-m05
pod_spec:
  cpu_size: 0.1
  mem_size: 200Mi

replicas:
  Search:
    frontend: 6
    reservation: 3
    search: 2
  Recommendation:
    frontend: 8
    profile: 5
    recommendation: 2

When I run main.py following problems occurs:

Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/consul_Deployment.yaml": deployments.apps "consul" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/consul_Service.yaml": services "consul" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/geo-pv_PersistentVolume.yaml": persistentvolumes "geo-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/geo-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "geo-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/jaeger_Deployment.yaml": deployments.apps "jaeger" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/jaeger_Service.yaml": services "jaeger" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-profile_Deployment.yaml": deployments.apps "memcached-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-profile_Service.yaml": services "memcached-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-rate_Deployment.yaml": deployments.apps "memcached-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-rate_Service.yaml": services "memcached-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-reserve_Deployment.yaml": deployments.apps "memcached-reserve" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-reserve_Service.yaml": services "memcached-reserve" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-geo_Deployment.yaml": deployments.apps "mongodb-geo" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-geo_Service.yaml": services "mongodb-geo" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-profile_Deployment.yaml": deployments.apps "mongodb-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-profile_Service.yaml": services "mongodb-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-rate_Deployment.yaml": deployments.apps "mongodb-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-rate_Service.yaml": services "mongodb-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-recommendation_Deployment.yaml": deployments.apps "mongodb-recommendation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-recommendation_Service.yaml": services "mongodb-recommendation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-reservation_Deployment.yaml": deployments.apps "mongodb-reservation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-reservation_Service.yaml": services "mongodb-reservation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-user_Deployment.yaml": deployments.apps "mongodb-user" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-user_Service.yaml": services "mongodb-user" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/profile-pv_PersistentVolume.yaml": persistentvolumes "profile-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/profile-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "profile-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/rate-pv_PersistentVolume.yaml": persistentvolumes "rate-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/rate-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "rate-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/recommendation-pv_PersistentVolume.yaml": persistentvolumes "recommendation-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/recommendation-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "recommendation-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/reservation-pv_PersistentVolume.yaml": persistentvolumes "reservation-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/reservation-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "reservation-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/user-pv_PersistentVolume.yaml": persistentvolumes "user-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/user-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "user-pvc" not found
Traceback (most recent call last):
  File "main.py", line 60, in <module>
    full_init("social", 30628)
  File "/home/caohch1/Downloads/Erms-main/testing/testCollection.py", line 29, in full_init
    DEPLOYER.full_init(app, configs.GLOBAL_CONFIG.nodes_for_infra, port)
  File "/home/caohch1/Downloads/Erms-main/deployment/deployer.py", line 54, in full_init
    deploy_by_yaml(tmp_infra_path, True, self.namespace)
  File "/home/caohch1/Downloads/Erms-main/deployment/deployer.py", line 149, in deploy_by_yaml
    utils.create_from_yaml(api_client, f"{folder}/{file}")
  File "/home/caohch1/anaconda3/envs/erms/lib/python3.8/site-packages/kubernetes/utils/create_from_yaml.py", line 162, in create_from_yaml
    return create_with(yml_document_all)
  File "/home/caohch1/anaconda3/envs/erms/lib/python3.8/site-packages/kubernetes/utils/create_from_yaml.py", line 153, in create_with
    raise FailToCreateError(failures)
kubernetes.utils.create_from_yaml.FailToCreateError: Error from server (Not Found): {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"hotel-reserv\" not found","reason":"NotFound","details":{"name":"hotel-reserv","kind":"namespaces"},"code":404}

It seems that hotel-reserv cannot be deployed properly. I guess I didn't modify the configuration files (i.e., *-global.yaml and utils.py) correctly but I don't know how to fix it.

[Nick-LCY]

Hi, you need to create the namespace in kubernetes first. Try the following command and then run it again. kubectl create ns hotel-reserv

[caohch-1]

Hi, you need to create the namespace in kubernetes first. Try the following command and then run it again. kubectl create ns hotel-reserv

It works! However, I encounter new problems as follow:

(erms) xxx@xxx:~/Downloads/Erms-main$ python main.py
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/consul_Deployment.yaml": deployments.apps "consul" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/consul_Service.yaml": services "consul" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/geo-pv_PersistentVolume.yaml": persistentvolumes "geo-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/geo-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "geo-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/jaeger_Deployment.yaml": deployments.apps "jaeger" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/jaeger_Service.yaml": services "jaeger" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-profile_Deployment.yaml": deployments.apps "memcached-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-profile_Service.yaml": services "memcached-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-rate_Deployment.yaml": deployments.apps "memcached-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-rate_Service.yaml": services "memcached-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-reserve_Deployment.yaml": deployments.apps "memcached-reserve" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/memcached-reserve_Service.yaml": services "memcached-reserve" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-geo_Deployment.yaml": deployments.apps "mongodb-geo" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-geo_Service.yaml": services "mongodb-geo" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-profile_Deployment.yaml": deployments.apps "mongodb-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-profile_Service.yaml": services "mongodb-profile" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-rate_Deployment.yaml": deployments.apps "mongodb-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-rate_Service.yaml": services "mongodb-rate" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-recommendation_Deployment.yaml": deployments.apps "mongodb-recommendation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-recommendation_Service.yaml": services "mongodb-recommendation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-reservation_Deployment.yaml": deployments.apps "mongodb-reservation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-reservation_Service.yaml": services "mongodb-reservation" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-user_Deployment.yaml": deployments.apps "mongodb-user" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/mongodb-user_Service.yaml": services "mongodb-user" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/profile-pv_PersistentVolume.yaml": persistentvolumes "profile-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/profile-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "profile-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/rate-pv_PersistentVolume.yaml": persistentvolumes "rate-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/rate-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "rate-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/recommendation-pv_PersistentVolume.yaml": persistentvolumes "recommendation-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/recommendation-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "recommendation-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/reservation-pv_PersistentVolume.yaml": persistentvolumes "reservation-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/reservation-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "reservation-pvc" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/user-pv_PersistentVolume.yaml": persistentvolumes "user-pv" not found
Error from server (NotFound): error when deleting "tmp/infra-hotel-reserv/user-pvc_PersistentVolumeClaim.yaml": persistentvolumeclaims "user-pvc" not found
Waiting for deployment finished...
Unfinished Pods: memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: consul-7d9cb8d-c5bnm, jaeger-5f7578ffbb-z4fcq, memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: consul-7d9cb8d-c5bnm, jaeger-5f7578ffbb-z4fcq, memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: jaeger-5f7578ffbb-z4fcq, memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: jaeger-5f7578ffbb-z4fcq, memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
Unfinished Pods: memcached-profile-78b8d7fb74-kpbfd, memcached-rate-54df8bc568-9bzgp, memcached-reserve-74b4d97df6-8cn4v, mongodb-geo-6d6959675-4pwzd, mongodb-profile-d74d4449-lnq86, mongodb-rate-5fb96448c6-fxk4n, mongodb-recommendation-64d9766d85-dx69x, mongodb-reservation-79fc8c5697-m4zqm, mongodb-user-5478769d8f-29284
...then just repeat...

I excute kubectl describe pod memcached-profile-78b8d7fb74-kpbfd -n hotel-reserv and it seems that I failed to pull the image:

Name:             memcached-profile-78b8d7fb74-kpbfd
Namespace:        hotel-reserv
Priority:         0
Service Account:  default
Node:             minikube-m05/192.168.49.6
Start Time:       Thu, 09 Mar 2023 18:34:15 +0800
...
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m50s                  default-scheduler  Successfully assigned hotel-reserv/memcached-profile-78b8d7fb74-kpbfd to minikube-m05
  Normal   Pulling    3m13s (x4 over 4m49s)  kubelet            Pulling image "k.harbor.siat.ac.cn/ms/memcached"
  Warning  Failed     3m12s (x4 over 4m49s)  kubelet            Failed to pull image "k.harbor.siat.ac.cn/ms/memcached": rpc error: code = Unknown desc = Error response from daemon: Get "https://k.harbor.siat.ac.cn/v2/": dial tcp: lookup k.harbor.siat.ac.cn on 192.168.65.2:53: no such host
  Warning  Failed     3m12s (x4 over 4m49s)  kubelet            Error: ErrImagePull
  Warning  Failed     3m1s (x6 over 4m49s)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m46s (x7 over 4m49s)  kubelet            Back-off pulling image "k.harbor.siat.ac.cn/ms/memcached"

Does there exist any other source for these images?

[Nick-LCY]

You can use these images from docker hub for now. I will upload our images to docker hub later and update the YAML files accordingly.

[caohch-1]

It works after I change the image source of mongodb, memcached, and ibench to docker-hub. However, something strange about the pod status happen. Since it is a different issue from current one, I open a new issue #3 . Hope you can discuss it with me if you have any idea.