search

tilt-dev / ctlptl

Making local Kubernetes clusters fun and easy to set up
Apache License 2.0
522 stars 38 forks source link

Using with podman #146

Open ccakes opened 2 years ago

ccakes commented 2 years ago

I'm trying to use this with podman using the Docker API-compat socket. It seems like it'll probably work but I've hit a snag and don't know enough about the Docker ecosystem to know the "right solution" for a PR.

Running ctlptl apply with config to create a registry fails. It creates the container but then can't find it to proceed with the workflow. The problem is this line

https://github.com/tilt-dev/ctlptl/blob/f5d2f384255a1a1d435f0ffddf17d1805d7fd828/pkg/registry/registry.go#L106

If I change that to use the fully-qualified path docker.io/library/registry:2 then it works fine. I'm unsure if that would also work on native Docker though, or if instead I should add a check for podman and set the filter depending on Docker v podman. Thoughts?

nicks commented 2 years ago

ooh nice! Ya, would love to accept a PR that gets this working with podman

I just tried that with the docker CLI and it seemed to work OK. We also have pretty good integration tests that will run on CI, so don't worry too much about breaking it.

nicks commented 2 years ago

@ccakes what version of podman are you using? i was playing around with this and couldn't get the podman container ls --filter to work at all :thinking: (on podman 3.0.1)

nicks commented 2 years ago

OK I upgraded to podman 3.4 and now see the problem. Sent out: https://github.com/tilt-dev/ctlptl/pull/new/nicks/issue146

Unfortunately, kind creation still blows up:

failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1

:\

ccakes commented 2 years ago

Sorry for the slow reply - I was using latest podman. I got about as far as you, I suspect that making it work for rootful containers is probably not that much work but rootless (how I was trying) would be much more difficult.

For rootless, it feels like the best option would be to create a pod and run both kind and the registry together in that but doing so would require more changes to ctlptl than I really feel comfortable tackling.

almereyda commented 1 year ago

Thanks to

we now have a working registry with Podman. Using Podman pods to create network contexts still appears out of scope.

Meanwhile it happens that a specific networking setup for K3d clusters on Podman would also be required for connecting a K3d registry to a K3d cluster (on Podman):

ncdc commented 1 year ago

I'm using ctlptl with podman but I had to create a config file in the podman VM to make it treat the registry created by ctlptl as insecure (otherwise I couldn't use it):

$ cat /etc/containers/registries.conf.d/100-kind.conf
[[registry]]
location = "localhost:61358"
insecure = true

A few questions:

aripollak commented 10 months ago

Besides the registries.conf addition (I put mine in ~/.config/containers/registries.conf), for running native rootless podman under Ubuntu 23.10 (upgraded from 22.04) I also had to install the golang-github-containernetworking-plugin-dnsname package so the kind container could resolve the ctlptl-registry hostname. Since I already had the kind network created in podman, I had to run podman network rm kind and recreate the cluster/registry with ctlptl after installing the -plugin-dnsname package.

I don't think ctlptl could/should install the plugin automatically, so that might just be something that just goes in the instructions, and it might also not be needed with a fresh Ubuntu install where you've only ever used podman 4.0+, since it then uses netavark by default, which has DNS support built in. Based on https://github.com/containers/podman/blob/main/docs/tutorials/basic_networking.md#bridge, the DNS lookup issue is also be fixable by destroying all podman resources under podman 4.0+ which should switch to netavark.

aaomidi commented 2 weeks ago 
Creating local k8s clusters
Creating registry "ctlptl-registry"...
registry.ctlptl.dev/ctlptl-registry created
reading docker-desktop settings: Get "http://localhost/app/settings": dial unix /Users/amir/Library/Containers/com.docker.docker/Data/backend.sock: connect: no such file or directory
make: *** [Makefile:120: dev-env-cluster] Error 1

Another fun little issue with podman. ctlptl assumes its docker-desktop and tries to do docker-desktop things to it.

nicks commented 2 weeks ago

@aaomidi ??? what command are you running? we have a lot of integration tests. the machines they run on don't even have docker-desktop installed...so that seems like an unrelated issue...