search

networkop / k8s-topo

Topology builder for network simulations inside K8S
BSD 3-Clause "New" or "Revised" License
72 stars 18 forks source link

HTTP response error when creating example #17

Closed crystalattice closed 2 years ago

crystalattice commented 2 years ago

I have tried several times to get the Usage example working but it fails every time, even on a clean VM of Ubuntu 21.10.

osboxes@osboxes:~$ k8s-topo --create examples/3node-host.yml
Traceback (most recent call last):
  File "/home/osboxes/.local/bin/k8s-topo", line 990, in <module>
    main()
  File "/home/osboxes/.local/bin/k8s-topo", line 842, in main
    uploaded = [bool(device.upload_to_k8s()) for _, device in devices.items()]
  File "/home/osboxes/.local/bin/k8s-topo", line 842, in <listcomp>
    uploaded = [bool(device.upload_to_k8s()) for _, device in devices.items()]
  File "/home/osboxes/.local/bin/k8s-topo", line 383, in upload_to_k8s
    response = CRD.create_namespaced_custom_object(
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/apis/custom_objects_api.py", line 178, in create_namespaced_custom_object
    (data) = self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/apis/custom_objects_api.py", line 264, in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api('/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'POST',
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 317, in call_api
    return self.__call_api(resource_path, method,
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 150, in __call_api
    response_data = self.request(method, url,
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 358, in request
    return self.rest_client.POST(url,
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 260, in POST
    return self.request("POST", url,
  File "/home/osboxes/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '22e55258-ac1d-41b8-b8c3-2f939fc0f6c1', 'X-Kubernetes-Pf-Prioritylevel-Uid': '683b060d-46ca-4227-b0b3-c8146f9df713', 'Date': 'Thu, 20 Jan 2022 18:55:26 GMT', 'Content-Length': '19'})
HTTP response body: 404 page not found
networkop commented 2 years ago

it looks like you're running this outside of the k8s cluster, right? do you have kubeconfig in ~/.kube/config? are you able to run kubectl commands from the same shell?

crystalattice commented 2 years ago

First, I just want to note that I'm new to k8s and literally learning it while I'm having to create virtual routers in k8s for work, so there's a good chance failures are not due to the code.

That said, I disabled the API exception error in kubernetes/client/rest.py and that let me go ahead. Not the best way to fix it but it helped me move forward.

osboxes@osboxes:~$ k8s-topo --create examples/3node-host.yml
INFO:__main__:All topology data has been uploaded
INFO:__main__:All pods have been created successfully
INFO:__main__:All pods have their TCP/443 port published starting from 30001
INFO:__main__:
 alias host-1='kubectl exec -it host-1 sh'
 alias host-2='kubectl exec -it host-2 sh'
 alias host-3='kubectl exec -it host-3 sh'

I also believe I ran the command within the cluster and received the same result:

osboxes@osboxes:/tmp/k8s-topo$ kubectl exec -it deployment/k8s-topo sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

/k8s-topo # k8s-topo --create examples/3node-host.yml
Traceback (most recent call last):
  File "/k8s-topo/bin/k8s-topo", line 983, in <module>
    main()
...

I do have a config file in ~/.kube/, but not ~/.kube/config/kubeconfig. I am able to run kubectl commands.

crystalattice commented 2 years ago

I don't know if it matters, but when I execute 

osboxes@osboxes:~$ kubectl exec -it deployment/k8s-topo sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/k8s-topo # k8s-topo --create examples/3node-host.yml

(after waiting for it to spin up, because I forgot about that part), I see that the create command is trying to use /usr/lib/python3.7, but I have Python 3.9 as the default.

From what I can tell, Python 3.7 is a snap package that was auto-installed at some point.

crystalattice commented 2 years ago

I commented out the exception lines in the Python 3.7 rest.py file and was able to continue. However, I couldn't complete the Usage example.

/k8s-topo # k8s-topo --create examples/3node-host.yml
INFO:__main__:All topology data has been uploaded
INFO:__main__:Not all pods have been created
INFO:__main__:All pods have their TCP/443 port published starting from 30001
INFO:__main__:
 alias host-1='kubectl exec -it host-1 sh'
 alias host-2='kubectl exec -it host-2 sh'
 alias host-3='kubectl exec -it host-3 sh'
/k8s-topo # k8s-topo --show examples/3node-host.yml
host-1@kind-worker
host-2@kind-worker2
host-3@kind-worker2
/k8s-topo # kubectl exec -it host-1 -- ping -c 1 12.12.12.2
error: unable to upgrade connection: container not found ("host-1")
/k8s-topo # kubectl exec -it host-1 -- ping -c 1 12.12.12.2
error: unable to upgrade connection: container not found ("host-1")
/k8s-topo # kubectl exec -it host-1 -- ping -c 1 13.13.13.3
error: unable to upgrade connection: container not found ("host-1")
networkop commented 2 years ago

I think disabilng that exception is not a solution. if you get 404 from k8s API server, that means it can't find that resource. I still don't fully understand you setup. Can you explain a bit where you cluster is, how you installed meshet-cni and k8s-topo, the more details the better.

crystalattice commented 2 years ago

I'm using kind for the k8s environment. 

osboxes@osboxes:~/meshnet-cni$ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   16h   v1.21.1
kind-worker          Ready    <none>                 16h   v1.21.1
kind-worker2         Ready    <none>                 16h   v1.21.1

I installed k8s-topo via Python pip and ran the command k8s-topo --create examples/3node-host.yml.

However, from your question, I realize that I neglected to install meshnet-cni first. I tried to remedy that by cloning meshnet-cni and, since I already have a kind cluster, I moved to the make docker command from the meshnet-cni instructions. That errored out with the following message:

Creating docker image networkop/meshnet:4bf3db7
unknown flag: --use
...
make: *** [Makefile:31: docker] Error 125
networkop commented 2 years ago

you don't need to build it, you can just run make install and it will pull the containers from dockerhub or you can just run kubectl apply -k manifests/base to install meshnet into your kind cluster.

crystalattice commented 2 years ago

Okay, it seems like I got everything working, but I can't view the network plot. Here's my commands:

osboxes@osboxes:~/meshnet-cni$ k8s-topo --show ~/k8s-topo/examples/3node-host.yml
host-1@kind-worker
host-2@kind-worker2
host-3@kind-worker2
osboxes@osboxes:~/meshnet-cni$ kubectl exec -it host-1 -- ping -c 1 12.12.12.2
kubectl exec -it host-1 -- ping -c 1 13.13.13.3
kubectl exec -it host-2 -- ping -c 1 23.23.23.3
Defaulted container "host-1" out of: host-1, init-host-1 (init)
PING 12.12.12.2 (12.12.12.2): 56 data bytes
64 bytes from 12.12.12.2: seq=0 ttl=64 time=0.356 ms

--- 12.12.12.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.356/0.356/0.356 ms
Defaulted container "host-1" out of: host-1, init-host-1 (init)
PING 13.13.13.3 (13.13.13.3): 56 data bytes
64 bytes from 13.13.13.3: seq=0 ttl=64 time=0.183 ms

--- 13.13.13.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.183/0.183/0.183 ms
Defaulted container "host-2" out of: host-2, init-host-2 (init)
PING 23.23.23.3 (23.23.23.3): 56 data bytes
64 bytes from 23.23.23.3: seq=0 ttl=64 time=0.107 ms

--- 23.23.23.3 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.107/0.107/0.107 ms

but when I run k8s-topo --graph ~/k8s-topo/examples/3node-host.yml I get an error on the web page Unable to connect Firefox can’t establish a connection to the server at 127.0.1.1:30000.

Did I misconfigure something?

networkop commented 2 years ago

I think this only works if you deploy k8s-topo as a pod inside the cluster. The pod should contain nginx https://github.com/networkop/k8s-topo/blob/848ea5bdaafa275b0f4c97706b7d14aa10af499f/Dockerfile#L31 and this is what's responsible for rendering of the graph.

crystalattice commented 2 years ago

Okay. That's not necessary for the application to work, just a cosmetic nice-to-have. Since I was able to get past the issues I had, I'll consider this issue resolved. Thanks for the help.