Is it possible to communicate with Kubernetes service through its DNS?

[Shaked]

I have been playing with buildkit and kubernetes as I'm looking for a solution to build docker images on top of k8s (in scale) especially due to the latest k8s changes regarding the docker daemon.

I'm curious to know if it is possible to do something like --addr tcp://serviceName.namespace.svc.cluster.local:1234 , i.e:

$ kubectl port-forward service/buildkitd 1234
$ buildctl \
  --addr tcp://serviceName.namespace.svc.cluster.local:1234 \
  --tlscacert .certs/client/ca.pem \
  --tlscert .certs/client/cert.pem \
  --tlskey .certs/client/key.pem \
  build --frontend dockerfile.v0 --local context=/path/to/dir --local dockerfile=/path/to/dir

The idea here is to be able to communicate with the deployment & service example through its service DNS instead of using a specific pod nam with kube-pod

Within an automdated CICD setup, this would make a lot of sense, especially when there's a need to scale the workers as much as possible without editing the build and release process (i.e: without changing the --addr:... configuration).

Thank you Shaked

[tonistiigi]

Yes, this is possible but you have to configure buildkitd to start on that tcp/tls address. By default, it listens on unix socket.

[Shaked]

Yes, this is possible but you have to configure buildkitd to start on that tcp/tls address. By default, it listens on unix socket.

I have used https://github.com/moby/buildkit/blob/master/examples/kubernetes/deployment%2Bservice.privileged.yaml#L20-L30

Do I need to change it? Any chance you could elaborate?

Thank you!

[tonistiigi]

Did you hit any issues with it? Or is your issue that you can't access serviceName.namespace.svc.cluster.local hostname from your local machine?

[Shaked]

@tonistiigi

Did you hit any issues with it? Or is your issue that you can't access serviceName.namespace.svc.cluster.local hostname from your local machine?

I'm trying to access through another pod. I tried both with kube-pod and tcp:

root@azure-devops-agent-dod-f5868db87-wdt77:/tmp# buildctl --debug --addr kube-pod://buildkitd.buildkit.svc.cluster.local:1234 build  --frontend dockerfile.v0 --local context=/tmp --local dockerfile=/tmp
DEBU[0000] commandconn: starting kubectl with [--context= --namespace= exec --container= -i buildkitd.buildkit.svc.cluster.local -- buildctl dial-stdio]
DEBU[0000] commandconn (kubectl):Error from server (NotFound): pods "buildkitd.buildkit.svc.cluster.local" not found
DEBU[0000] stopping session
[+] Building 0.0s (0/0)
error: failed to get status: rpc error: code = Unavailable desc = connection closed
28657 v0.8.3 buildctl --debug --addr kube-pod://buildkitd.buildkit.svc.cluster.local:1234 build --frontend dockerfile.v0 --local context=/tmp --local dockerfile=/tmp
github.com/moby/buildkit/client.(*Client).solve.func4
    /src/client/solve.go:249
golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1357

root@azure-devops-agent-dod-f5868db87-wdt77:/tmp# buildctl --debug --addr tcp://buildkitd.buildkit.svc.cluster.local:1234 build  --frontend dockerfile.v0 --local context=/tmp --local dockerfile=/tmp
DEBU[0000] stopping session
[+] Building 0.0s (0/0)
error: failed to get status: rpc error: code = Unavailable desc = connection closed
28681 v0.8.3 buildctl --debug --addr tcp://buildkitd.buildkit.svc.cluster.local:1234 build --frontend dockerfile.v0 --local context=/tmp --local dockerfile=/tmp
github.com/moby/buildkit/client.(*Client).solve.func4
    /src/client/solve.go:249
golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1357

EDIT:

telnet seems to work fine, so the host is available

telnet> root@azure-devops-agent-dod-f5868db87-wdt77:/tmp# telnet buildkitd.buildkit.svc.cluster.local 1234
Trying 10.0.112.74...
Connected to buildkitd.buildkit.svc.cluster.local.
Escape character is '^]'.

[tonistiigi]

kube-pod://buildkitd.buildkit.svc.cluster.local:1234 is wrong. Argument for kube-pod is the pod name, not tcp address.

Not sure about the second one. @AkihiroSuda

[Shaked]

@michalzxc mentioned a similar thing in https://github.com/moby/buildkit/issues/2136#issuecomment-848600739 maybe it's possible?

[michalzxc]

helm template helm-charts/buildkitd --namespace drone-ci-exec                                                                           
---
# Source: buildkitd/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: buildkitd
  name: buildkitd
spec:
  ports:
    - port: 1234
      protocol: TCP
  selector:
    app: buildkitd
---
# Source: buildkitd/templates/statefullset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: buildkitd
  name: buildkitd
  namespace: drone-ci-exec
spec:
  serviceName: buildkitd
  podManagementPolicy: Parallel
  replicas: 2
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      labels:
        app: buildkitd
    spec:
      tolerations:
        - effect: NoSchedule
          key: droneciexec
          operator: Equal
          value: "true"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/role
                operator: In
                values:
                - droneciexec
      containers:
        - name: buildkitd
          image: moby/buildkit:master
          args:
            - --addr
            - unix:///run/buildkit/buildkitd.sock
            - --addr
            - tcp://0.0.0.0:1234
          volumeMounts:
          - mountPath: /var/lib/buildkit
            name: imgvolume
          readinessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          livenessProbe:
            exec:
              command:
                - buildctl
                - debug
                - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          ports:
            - containerPort: 1234
          securityContext:
            privileged: true
      volumes:
        - name: imgvolume
          hostPath:
            path: /var/lib/drone-img-cache/
            type: Directory

And client end is docker with:

#buildkit
ADD buildkit-v0.8.2.linux-amd64.tar.gz /usr/local/
ENV BUILDKIT_HOST tcp://buildkitd.drone-ci-exec.svc.cluster.local:1234
ENV DOCKER_CONFIG /root/.docker/
RUN mkdir -p /root/.docker/
ADD config.json /root/.docker/config.json
#/buildkit

And it all works

Running with

buildctl build --frontend=dockerfile.v0 --local dockerfile=. --local context=.

in a directory where is "Dockerfile" it doesn't seems to support custom names for Dockerfile

[Shaked]

@michalzxc - thank you!

@tonistiigi so with the help of @michalzxc, I figured that the problem was that I didn't use the client certificates, i.e:

BUILDKIT_HOST=tcp://buildkitd.dod-temp.svc.cluster.local:12345 buildctl --tlscacert /azp/client/ca.pem --tlscert /azp/client/cert.pem --tlskey /azp/client/key.pem --debug  build  --frontend dockerfile.v0 --local context=/tmp --local dockerfile=/tmp

Not sure if it's possible, but maybe it would be best if the error message for a daemon that requires certificate would be more descriptive, instead of:

error: failed to get status: rpc error: code = Unavailable desc = connection closed

I'd be happy to help if you can point me where this can be handled.

Shaked