search

linkerd / linkerd2

Ultralight, security-first service mesh for Kubernetes. Main repo for Linkerd 2.x.
https://linkerd.io
Apache License 2.0
10.62k stars 1.27k forks source link

Add check to ensure opaque ports is set on service if also set on pod #6177

Closed mateiidavid closed 3 years ago

mateiidavid commented 3 years ago

This is based on #6160: if an opaque-ports is set on a workload (e.g a pod template in a Deployment) but not on its corresponding service, then the opaque_transport is correctly set on the endpoint however we still do protocol detection. This results in an unexpected Opaque Transport header picked up by the inbound proxy (concluding with a closed conn):

[    19.551490s] DEBUG ThreadId(01) inbound:accept{client.addr=10.42.0.100:39078}:server{port=4143}:direct: linkerd_transport_header::server: Read transport header header=TransportHeader { port: 8080, name: None, protocol: Some(Http2) }
[    19.551569s]  INFO ThreadId(01) inbound:accept{client.addr=10.42.0.100:39078}: linkerd_app_core::serve: Connection closed error=a named target must be provided on gateway connections

A simple way to mitigate this problem is to add a new check for opaque-ports annotation and make sure it exists on a service if it also exists on a workload. An exception to this should be headless services (where clusterIP: None) whose corresponding workloads may have the annotation -- endpoints for these types of services are handled differently.

My suggestion for this would be to do something similar to #5718. We should ideally make this check part of linkerd check --proxy which is namespace specific. Then, for each check, we would look at the services in that namespace, check one of the underlying pods, if it has the config.linkerd.io/opaque-ports annotation but the service doesn't, print an error/warning.

Note: non-meshed services and workloads may still have the opaque-ports annotation.

migue commented 3 years ago

👋 if you're okay with it I would like to give it a try!! I will probably ask some [dumb] questions since this is my first time around this ...

migue commented 3 years ago

I've opened this [draft] pull https://github.com/linkerd/linkerd2/pull/6192

Not sure if I've taken the right path; I'd appreciate any hint before I move forward. The tests are missing but I've tested my local build against my a Kubernetes setup where a bunch of services are missing the annotation:

linkerd-opaque-ports-definition
-------------------------------
× the opaque-ports labels have been properly defined
    config.linkerd.io/opaque-ports annotation doesn't match for service kubernetes. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service actions-api-server. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service alive. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service aqueduct-lite. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service audit-rsyslog. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service authzd. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service babeld. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service elasticsearch. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service failbotg. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service gpgverify. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service hookshot-go. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service kafka-lite. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service launch-credz. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service launch-deployer. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service launch-receiver. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service lfs-server. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service memcached-0. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service memcached-1. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service memcached-2. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service minio. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service mps-api-server. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service nginx-unicorn. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service octoshift. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service packages. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service render. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service router. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service router-internal. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service runner-api-server. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service spokes-sweeper. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service spokesd. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service token-api-server. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service token-scanning-service. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service treelights. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service turboscan. Service annotation:   Pod annotation: 9480
config.linkerd.io/opaque-ports annotation doesn't match for service konnectivity. Service annotation:   Pod annotation: 9480
    see https://linkerd.io/2/checks/#linkerd-opaque-ports-definition for hints

Thanks a lot!