Services using named ports fail with error 'missing subsets for port'

[laverya]

When using a service definition like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-app
  name: my-service
  namespace: default
spec:
  ports:
  - name: my-port
    port: 9000
    targetPort: my-app-port
  selector:
    app: my-app
  type: ClusterIP

I get this error time="2019-01-03T01:33:43Z" level=error msg="translator service issue on ingress: default/my-argo-tunnel, host: example.com, path: {Path:/ Backend:{ServiceName:my-service ServicePort:{Type:0 IntVal:9000 StrVal:}}}, err: \"endpoints 'default/my-service' missing subsets for port 'my-app-port'\""

When using a literal targetPort (like targetPort: 8000) things work fine.

[mattalberts]

The Service posted doesn't match the error message, it's looking for a port named http-private. Was the Service posted used to generate the error?

[mattalberts]

I've verified the behavior with the following manifests

apiVersion: v1
kind: Service
metadata:
  labels:
    app: echo
  name: echo
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: echo
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: echo
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echo
        image: k8s.gcr.io/echoserver:1.10
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
      terminationGracePeriodSeconds: 60
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: argo-tunnel
  labels:
    ingress: argo-tunnel
  name: echo
spec:
  tls:
  - hosts:
    - echo.mydomain.com
    secretName: mydomain.com
  rules:
  - host: echo.mydomain.com
    http:
      paths:
      - backend:
          serviceName: echo
          servicePort: http

tip: change mydomain.com to your domain

The tunnel starts as expected

time="2019-01-03T14:45:48Z" level=error msg="translator service issue on ingress: default/echo, host: echo.mattalberts.pink, path: {Path: Backend:{ServiceName:echo ServicePort:{Type:1 IntVal:0 StrVal:http}}}, err: \"endpoints 'default/echo' missing subsets for port 'http'\""
time="2019-01-03T14:45:48Z" level=debug msg="router update route: default/echo"
time="2019-01-03T14:45:52Z" level=debug msg="translator update service: default/echo"
time="2019-01-03T14:45:52Z" level=debug msg="translator attach tunnel: default/echo, rule: {service:{name:echo namespace:default} secret:{name:mattalberts.pink namespace:default} host:echo.mattalberts.pink port:80}"
time="2019-01-03T14:45:52Z" level=debug msg="router update by service: default/echo"
time="2019-01-03T14:45:52Z" level=debug msg="router update route: default/echo"
time="2019-01-03T14:45:52Z" level=info msg="link start host: echo.mattalberts.pink, origin: echo.default:80"
time="2019-01-03T14:45:52Z" level=debug msg="Compression is not supported"
time="2019-01-03T14:45:52Z" level=debug msg="initiating RPC stream to register"

tip: the error "endpoints 'default/echo' missing subsets for port 'http'" will temporarily display as the system waits for endpoints to be available (e,g, the service backing pods become available on the port specified).

The reported behavior is not an issue, please verify your example (specifically the port http-private)

[laverya]

Looks like I copied the error message from when I first saw the issue and the service from the reproduction... whoops. Fixed it

In any case, I'll take a look at our setup and see what might be leading to this, because we certainly saw "tunnel doesn't work with named targetPort, does work with raw port #" on multiple occasions. (The service worked either way)

[smartyjohn]

I can replicate this error in v0.6.2.

time="2019-01-07T21:47:46Z" level=error msg="translator service issue on ingress: live/live-ing, host: echo.[domain removed], path: {Path: Backend:{ServiceName:echo-svc ServicePort:{Type:1 IntVal:0 StrVal:echo-http}}}, err: \"endpoints 'live/echo-svc' missing subsets for port 'http'\""

The Ingress is forwarding to a backend with:

      - backend:
          serviceName: echo-svc
          servicePort: echo-http

The Service defines its ports as:

  ports:
    - port: 80
      targetPort: http
      name: echo-http

The Deployment configures it's ports as (used in the targetPort lookup):

          ports: # required for gce L7 load balancers to find readinessProbe
            - containerPort: 8080
              name: http
            - containerPort: 8443
              name: https

The service-generated Endpoint has the following data:

apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: 2019-01-03T21:25:07Z
  labels:
    app: echo
    package: util
    tier: operations
  name: echo-svc
  namespace: live
  resourceVersion: "276767"
  selfLink: /api/v1/namespaces/live/endpoints/echo-svc
  uid: 0b3c2c42-0f9e-11e9-ba49-363354237dd9
subsets:
- addresses:
  - ip: 10.2.1.6
    nodeName: spc62r36k7-worker-1
    targetRef:
      kind: Pod
      name: echo-deploy-6b7564c5b4-8kr46
      namespace: live
      resourceVersion: "8221"
      uid: 0b2ceaea-0f9e-11e9-ba49-363354237dd9
  - ip: 10.2.3.6
    nodeName: spc62r36k7-worker-13886003
    targetRef:
      kind: Pod
      name: echo-deploy-6b7564c5b4-wswj9
      namespace: live
      resourceVersion: "276766"
      uid: fb63c04d-104e-11e9-ba49-363354237dd9
  ports:
  - name: echo-http
    port: 8080
    protocol: TCP

The Endpoint data appears correct, with the proper subsets defined. However it appears the controller code is using the wrong name for the port lookup in the subset data where the proper port name is the one from Ingress servicePort and defined in Service.ports[].name -- that is echo-http. Instead, it seems the Endpoint query is (incorrectly) using the Service.ports[].targetPort and/or the Deployment/Pod's port name or http in this case, as displayed in the original error.

As @laverya mentioned, using a numeric Service.ports[].targetPort seems to bypass the issue.

[mattalberts]

Hmmm! Thanks for reporting! Looks like I’ll have to test it :)

[mattalberts]

This should be the way forward :)

[acrogenesis]

I get this error sometimes (without changing the ingress config).

time="2019-06-29T19:18:29Z" level=error msg="translator service issue on ingress: default/exp626-sname, host: sname.domain.com, path: {Path: Backend:{ServiceName:exp626-sname ServicePort:{Type:1 IntVal:0 StrVal:http}}}, err: \"endpoints 'default/exp626-sname' missing subsets for port 'http'\""

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: argo-tunnel
  labels:
    ingress: argo-tunnel
  name: exp626-sname
spec:
  tls:
  - hosts:
    - sname.domain.com
    secretName: domain.com
  rules:
  - host: sname.domain.com
    http:
      paths:
      - backend:
          serviceName: exp626-sname
          servicePort: http