Kubernetes VM don't have internet connection

[raihanjp]

ISSUE TYPE

• Bug Report
• Improvement Request

COMPONENT NAME

CloudStack Kubernetes Provider

CLOUDSTACK VERSION

4.18.1

OS / ENVIRONMENT

Ubuntu 22.04

SUMMARY

Can't pull any image

Failed to pull image "registry.k8s.io/metrics-server/metrics-server:v0.6.4": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/metrics-server/metrics-server:v0.6.4": failed to resolve reference "registry.k8s.io/metrics-server/metrics-server:v0.6.4": failed to do request: Head "https://registry.k8s.io/v2/metrics-server/metrics-server/manifests/v0.6.4": dial tcp: lookup registry.k8s.io on 8.8.8.8:53: read udp 10.1.1.231:39706->8.8.8.8:53: i/o timeout

Ping from control node cloud@skripsi-control-18c618bc8bd:$ ping -c 1 google.com ping: unknown host cloud@skripsi-control-18c618bc8bd:$ ping -c 1 registry.k8s.io ping: unknown host

STEPS TO REPRODUCE

Fresh kubernetes cluster installation Kubernetes version: https://download.cloudstack.org/testing/cks/setup-1.27.8.iso kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

EXPECTED RESULTS

metrics-server-5d875656f5-flz97                       1/1     Running 

ACTUAL RESULTS

metrics-server-5d875656f5-flz97                       0/1     ImagePullBackOff

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

@raihanjp

which type of zone (basic/advanced/advanced with security group) and network (isolated, vpc tier, shared) do you use ?

can you run the same ping commands from CloudStack virtual router of the network ?

[raihanjp]

@weizhouapache -I'm using advanced zone and DefaultNetworkOfferingforKubernetesService which is isolated.

-Still can't ping when i ran the same ping commands on the virtual router of the network

[raihanjp]

I installed cloudstack exactly following this tutorial https://rohityadav.cloud/blog/cloudstack-kvm/ adapt with my network configuration

[weizhouapache]

@weizhouapache -I'm using advanced zone and DefaultNetworkOfferingforKubernetesService which is isolated.

-Still can't ping when i ran the same ping commands on the virtual router of the network

@raihanjp what's the default egress policy ? is there any egress rules ? can you ping the gateway of public IPs in the virtual router ?

[raihanjp]

@weizhouapache

• The default egress policy is allow and there's no any egress rules
• I can't ping the gateway of the public IP, but i can ping 8.8.8.8

[weizhouapache]

@weizhouapache

• The default egress policy is allow and there's no any egress rules
• I can't ping the gateway of the public IP, but i can ping 8.8.8.8

@raihanjp all looks good. now the problem is, you can ping 8.8.8.8, but not for google.com, right ?

[raihanjp]

@weizhouapache Yes i can't ping google.com or registry.k8s.io or any domain. I want to troubleshoot this because i tought it will solve the problem i'm facing that is failed to pull image

[weizhouapache]

@weizhouapache Yes i can't ping google.com or registry.k8s.io or any domain. I want to troubleshoot this because i tought it will solve the problem i'm facing that is failed to pull image

It looks like a DNS issue Do you use 8.8.8.8 as the dns server ?

[raihanjp]

Yes, i use 8.8.8.8 as the dns server

[raihanjp]

if my systemvm can ping google.com and registry.k8s.io, the problem must be on the kubernetes cluster creation process right? or do you think there could be a misconfiguration on the apache cloudstack installation?

[weizhouapache]

if my systemvm can ping google.com and registry.k8s.io, the problem must be on the kubernetes cluster creation process right? or do you think there could be a misconfiguration on the apache cloudstack installation?

ping/dns should work in the CloudStack virtual router (it is not in the scope of CKS configuration)

can you run more commands in the VR ?

cat /etc/resolv.conf
nslookup google.com

[raihanjp]

the resolv.conf only have this and nslookup can't ping google.com

nameserver 8.8.8.8

i added

nameserver 192.168.10.1

finally the VR can ping and nslookup google.com, but the VR itself is not pingable and it can't ping it's own ip address. and the kubernetes still can't pull the image, i've tried with other image and it still has the same error. (s-1-vm = system VM & r-11-VM = VR)

Does this ip route on the system vm and the VR matter? the VR name changed from the last image i post after i reboot the VR

[weizhouapache]

@raihanjp you can add 192.168.10.1 as a dns server in the zone setting.

when you add nameserver in /etc/resolv.conf manually , please restart dnsmasq service in VR and retry.

[raihanjp]

• do you mean this by adding the dns server in the zone setting? And should the use.external.dns in the global setting set to "true"?
• i've restart dnsmasq service in VR, i just noticed if i reboot the VR, the config on resolv.conf is back to default only "nameserver 8.8.8.8"

[weizhouapache]

• do you mean this by adding the dns server in the zone setting? And should the use.external.dns in the global setting set to "true"?
• i've restart dnsmasq service in VR, i just noticed if i reboot the VR, the config on resolv.conf is back to default only "nameserver 8.8.8.8"

@raihanjp for existing networks, you can update the DNS1 and DNS2 in the networt details.

Did it work after you added nameserver and restarted dnsmasq ?

[raihanjp]

@weizhouapache after adding the nameserver and restart dnsmasq it just work when pinging google.com from the VR, but the kubernetes vm still can't ping google.com. the configuration of the resolv.conf revert to the default too thats "nameserver 8.8.8.8" only, if the VR is rebooted.

Currently i'm reinstalling my ubuntu (including cloudstack management), because previously I had successfully used a Kubernetes cluster without problems, but something happened that caused the server to have to be factory reset. I'll get back to you with the update. Thankyou so muchweizhou

[raihanjp]

@weizhouapache hello, i'm here to update the progress. so last night i reinstall ubuntu and cloudstack management, last night the problem is still the same, i rerun the same process to reproduce the problem. I try creating the k8s cluster using this ISO https://download.cloudstack.org/testing/cks/, i tried both 1.27.8 and 1.28.4 and faced the same problem, the k8s node can't ping google.com and the VR can't ping google.com too, i didn't change any settings at all, so i turned off the server last night. this morning when i turned on the server i didn't face any problem and i didn't change any settings. Now i can ping google.com from my VR and VM, and my VM can already pull any image. I really don't know where the problem is. Thank you very much for your help

[weizhouapache]

@weizhouapache hello, i'm here to update the progress. so last night i reinstall ubuntu and cloudstack management, last night the problem is still the same, i rerun the same process to reproduce the problem. I try creating the k8s cluster using this ISO https://download.cloudstack.org/testing/cks/, i tried both 1.27.8 and 1.28.4 and faced the same problem, the k8s node can't ping google.com and the VR can't ping google.com too, i didn't change any settings at all, so i turned off the server last night. this morning when i turned on the server i didn't face any problem and i didn't change any settings. Now i can ping google.com from my VR and VM, and my VM can already pull any image. I really don't know where the problem is. Thank you very much for your help

@raihanjp some changes might be applied after rebooting For example, when you update the DNS setting of the network, you need to restart the VR to make the config effective.

[raihanjp]

@weizhouapache Here to update again. I kept facing the same issue when creating a new k8s cluster. I've solved the issue that works everytime for me. Before, the guest network only use DNS1 = 8.8.8.8

To solve the issue, i add the dns on the guest network so it use DNS 1=1.1.1.1 and DNS 2=8.8.8.8