Disappeared service endpoints

[paivin-dn]

What happened?

I tried to set up proxying from the k8s cluster to external service using Pomerium ingress controller + service + endpoint. My manifests look like:

apiVersion: v1
kind: Service
metadata:
  name: mysvc
  labels:
    app: mysvc
spec:
  selector:
    app: mysvc
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

apiVersion: v1
kind: Endpoints
metadata:
  name: mysvc
  labels:
    app: mysvc
subsets:
  - addresses:
      - ip: 10.101.10.5
    ports:
      - name: web
        port: 80
        protocol: TCP

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tech-docs
  annotations:
    ingress.pomerium.io/policy: |
      - allow:
          or:
            - domain:
                is: example.com
spec:
  ingressClassName: pomerium
  rules:
    - host: myhost.example.com
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: mysvc
                port:
                  number: 80

After deploying it works, but several hours later I see that the IP address in the Endpoint was deleted.

What did you expect to happen?

The IP in the Endpoint should exist.

How'd it happen?

1. Ran x
2. Clicked y
3. Saw error z

What's your environment like?

• cloud k8s in digitlaocen, pomerium deployed 0.25.0

What's your config.yaml?

apiVersion: ingress.pomerium.io/v1
kind: Pomerium
metadata:
  name: global
spec:
  secrets: "pomerium/bootstrap"
  authenticate:
    url: "https://myayth.example.com"
  storage:
    postgres:
      secret: "pomerium/storage"
  identityProvider:
    provider: auth0
    secret: pomerium/provider"
    url: "auth0_url"
  certificates:
    - "pomerium/cert"

What did you see in the logs?

"level":"warn","time":"2024-07-18T20:39:37Z","msg":"stapling OCSP","service":"autocert","error":"unable to write OCSP staple file for [*.example.com example.com]: mkdir /.local: read-only file system"

[wasaga]

1. Pomerium does not modify your manifests, it only listens to changes. That is reflected in the supplied RBAC https://github.com/pomerium/ingress-controller/blob/main/config/pomerium/rbac/role.yaml
2. if your intent is to reference the Service that is running within your cluster, you do not normally need create Endpoints manifest. Your kubernetes would manage that based on currently running pods.
3. I'm a bit confused with "external service". if your intent is to front an application that is running outside of your kubernetes cluster, that should be achieved by creating an External Name Service without selectors or using headless service

[paivin-dn]

I have the following schema

After deploying the endpoint for several hours, the IP address in the endpoint(that points to the VM IP address) disappeared. And it happened regularly. I don't want to use the External Name Service because my internal service may don't have DNS record

[wasaga]

you should be able to put an IP address into External Name Service. You do not need to create an Endpoint document yourself.

Normally, Endpoint is not manually created, it is updated by kubernetes itself based on your Service type and observed Pods:

The control plane automatically creates EndpointSlices for any Kubernetes Service that has a selector specified.