Can colima with k3s and a nginx-ingress point to locally hosted service?

[aelsnz]

Description

Deploy colima with kubernetes enabled, I then normally would deploy a service as a POD, expose the service via ingress using nginx-ingress. This works well.

Now the problem/question I am trying to solve is having my ingress point to a service that is external running on my localhost.

Example:

The service I create runs on lets say port 8080 and what I want to do is run it on my local host on port 8080 and code/debug locally - not running the service in a pod; but ideally still being able to access this via the ingress URL - example: demo.example.com which should then direct to my service on localhost port 8080.

But I get "Bad Gateway" as my ingress cannot get to the correct destination. And what seem to happen is the traffic is hitting the ingress External-IP - 192.168.106.10 on port 8080, but this will not work as it will not get to my local host.

$ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.43.254.45   192.168.106.10   80:31912/TCP,443:30173/TCP   23h

Environment is on Apple Mac M1 using Colima 0.5.4 with command below to create the environment:

colima --profile local-demo start \
--cpu 6 \
--memory 8 \
--disk 20 \
--with-kubernetes \
--kubernetes-disable="traefik" \
--kubernetes-version="v1.24.3+k3s1" \
--network-address \
--ssh-agent \
--mount-type 9p

$ colima list
PROFILE      STATUS     ARCH       CPUS    MEMORY    DISK     RUNTIME       ADDRESS
local-demo    Running    aarch64    6       8GiB      20GiB    docker+k3s    192.168.106.10

Is this possible with colima with kubernetes? Is tunnels needed like what is required in minikube with "minikube tunnel" etc?

I will work on creating a basic example to post as my current one is getting complex, but wanted to confirm if this is even possible. Any pointers much appreciated.

[abiosoft]

I think the services should be able to access your host at host.docker.internal

[aelsnz]

Thank you for your input, but I am not sure I understand how I would use that if you can maybe provide more detail on how I would use host.docker.internal?

In my case I need to access use the ingress url - demo.example.com and then have this direct to service that runs in localhost on my Mac, outside of colima/k3s. I am using one service as example on localhost:8080 but I do have other services also in the ingress configuration - depending on the URL it will be directed to the required port/service.
I hope this make sense.

                        / ---> demo.example.com/hello  --> to point to the external service - localhost:8080
-----> ingress ngingx --
                        \ ---> demo.example.com/other --> points to another service (pod) in colima/k3s.

In example above, I guess my question comes down to, how can I get from ingress path demo.example.com/hello to the external service on localhost:8080

The advantage when doing something like this is that you can code/debug etc the service locally, but still have it as part of the overall application in a local setup using Colima/k3s.

[abiosoft]

Have you tried swapping localhost for host.docker.internal?

                        / ---> demo.example.com/hello  --> to point to the external service - host.docker.internal:8080
-----> ingress ngingx --
                        \ ---> demo.example.com/other --> points to another service (pod) in colima/k3s.

[aelsnz]

I will review, but not sure as I do not specify "localhost" anywhere at moment. The service runs on the local mac on 0.0.0.0 port 8080 - and what I am trying to see is how can I get the Ingress to point to this. When the ingress is created I end up with:

$ kubectl get service ingress-nginx-controller --namespace=ingress-nginx
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx-controller   LoadBalancer   10.43.254.45   192.168.106.10   80:31912/TCP,443:30173/TCP   23h

And traffic is directed at 192.168.106.10 which then does not go anywhere as it ends up in the colima VM. The gateway for this range: 192.168.106.1 does route to the underlying host and if I try access my service using curl inside the VM on 192.168.106.1:8080 I get a a response.

[aelsnz]

Ok, interesting update, if I update the service definition and update the externalName to the "col" network gateway of 192.168.106.1 it works!

  type: ExternalName
  externalName: 192.168.106.1

[abiosoft]

In that case I'd recommend 192.168.5.2 instead.

It is constant and would still work if you change Colima's network driver.

[aelsnz]

Thank you - let me have a look at that and test that. Is there any documentation or notes you can point me at with regards to the networking setup and what IPs (networks) are used for what etc?

[abiosoft]

It is relatively undocumented.

There is something upstream on Lima https://github.com/lima-vm/lima/blob/master/docs/network.md#host-ip-19216852, but most of the information there is only useful if you are using Lima directly, Colima abstracts all the complexities to provide a more simple interface for the user.