search

dynatrace-oss / progressiveDelivery-masterclass

Apache License 2.0
4 stars 11 forks source link

Traces received from k6 but not real user traffic #99

Open agardnerIT opened 1 year ago

agardnerIT commented 1 year ago

I can see traces in Jaeger from the k6 traffic but no traces when I hit the demo app via a browser.

image

Flag value changes to the CR are respected and hte application responds accordingly - so the flags are working.

Impact: Observability not app functionality.

The OFO controller manager has these logs:

2023-08-17T00:18:39Z    INFO    controller-runtime.metrics  Metrics server is starting to listen    {"addr": "127.0.0.1:8080"}
2023-08-17T00:18:39Z    INFO    controller-runtime.builder  skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called   {"GVK": "core.openfeature.dev/v1alpha1, Kind=FeatureFlagConfiguration"}
2023-08-17T00:18:39Z    INFO    controller-runtime.builder  skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called {"GVK": "core.openfeature.dev/v1alpha1, Kind=FeatureFlagConfiguration"}
2023-08-17T00:18:39Z    INFO    controller-runtime.webhook  Registering webhook {"path": "/convert"}
2023-08-17T00:18:39Z    INFO    controller-runtime.builder  Conversion webhook enabled  {"GVK": "core.openfeature.dev/v1alpha1, Kind=FeatureFlagConfiguration"}
2023-08-17T00:18:39Z    INFO    controller-runtime.builder  skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called   {"GVK": "core.openfeature.dev/v1alpha1, Kind=FlagSourceConfiguration"}
2023-08-17T00:18:39Z    INFO    controller-runtime.builder  skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called {"GVK": "core.openfeature.dev/v1alpha1, Kind=FlagSourceConfiguration"}
2023-08-17T00:18:39Z    INFO    controller-runtime.builder  Conversion webhook enabled  {"GVK": "core.openfeature.dev/v1alpha1, Kind=FlagSourceConfiguration"}
2023-08-17T00:18:39Z    INFO    controller-runtime.webhook  Registering webhook {"path": "/mutate-v1-pod"}
2023-08-17T00:18:39Z    INFO    controller-runtime.webhook  Registering webhook {"path": "/validate-v1alpha1-featureflagconfiguration"}
2023-08-17T00:18:39Z    INFO    setup   starting manager
2023-08-17T00:18:39Z    INFO    setup   restoring flagd-kubernetes-sync cluster role binding subjects from current cluster state
2023-08-17T00:18:39Z    INFO    controller-runtime.webhook.webhooks Starting webhook server
2023-08-17T00:18:39Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-08-17T00:18:39Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
2023-08-17T00:18:39Z    INFO    controller-runtime.certwatcher  Updated current TLS certificate
2023-08-17T00:18:39Z    INFO    controller-runtime.webhook  Serving webhook server  {"host": "", "port": 9443}
2023-08-17T00:18:39Z    INFO    controller-runtime.certwatcher  Starting certificate watcher
I0817 00:18:39.360130       1 leaderelection.go:248] attempting to acquire leader lease open-feature-operator-system/131bf64c.openfeature.dev...
I0817 00:18:39.375002       1 leaderelection.go:258] successfully acquired lease open-feature-operator-system/131bf64c.openfeature.dev
2023-08-17T00:18:39Z    DEBUG   events  open-feature-operator-controller-manager-5db85c65b4-4vc4n_fb61fa13-7901-4865-808c-66f8766d5b2e became leader    {"type": "Normal", "object": {"kind":"Lease","namespace":"open-feature-operator-system","name":"131bf64c.openfeature.dev","uid":"d23c05f6-9e92-4230-8bfd-b7e7e2804e4f","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1767"}, "reason": "LeaderElection"}
2023-08-17T00:18:39Z    INFO    Starting EventSource    {"controller": "featureflagconfiguration", "controllerGroup": "core.openfeature.dev", "controllerKind": "FeatureFlagConfiguration", "source": "kind source: *v1alpha1.FeatureFlagConfiguration"}
2023-08-17T00:18:39Z    INFO    Starting EventSource    {"controller": "featureflagconfiguration", "controllerGroup": "core.openfeature.dev", "controllerKind": "FeatureFlagConfiguration", "source": "kind source: *v1.ConfigMap"}
2023-08-17T00:18:39Z    INFO    Starting Controller {"controller": "featureflagconfiguration", "controllerGroup": "core.openfeature.dev", "controllerKind": "FeatureFlagConfiguration"}
2023-08-17T00:18:39Z    INFO    Starting EventSource    {"controller": "flagsourceconfiguration", "controllerGroup": "core.openfeature.dev", "controllerKind": "FlagSourceConfiguration", "source": "kind source: *v1alpha1.FlagSourceConfiguration"}
2023-08-17T00:18:39Z    INFO    Starting Controller {"controller": "flagsourceconfiguration", "controllerGroup": "core.openfeature.dev", "controllerKind": "FlagSourceConfiguration"}
2023-08-17T00:18:39Z    INFO    Starting workers    {"controller": "featureflagconfiguration", "controllerGroup": "core.openfeature.dev", "controllerKind": "FeatureFlagConfiguration", "worker count": 1}
2023-08-17T00:49:57Z    DEBUG   controller-runtime.webhook.webhooks wrote response  {"webhook": "/validate-v1alpha1-featureflagconfiguration", "code": 200, "reason": "", "UID": "2c0fd772-20e4-45ce-b874-0f5e4fcac8b4", "allowed": true}
2023-08-17T00:49:57Z    INFO    FeatureFlagConfiguration Controller ReconcilingFeatureFlagConfiguration
2023-08-17T00:49:57Z    INFO    FeatureFlagConfiguration Controller Finished Reconciling FeatureFlagConfiguration
2023/08/17 00:49:57 http: TLS handshake error from 10.244.0.1:61265: EOF
2023-08-17T00:49:57Z    INFO    FlagSourceConfiguration Controller  Searching for FlagSourceConfiguration
2023-08-17T00:49:57Z    INFO    FlagSourceConfiguration Controller  Finished Reconciling FlagSourceConfiguration
2023/08/17 00:49:57 http: TLS handshake error from 10.244.0.1:31466: EOF
2023/08/17 00:49:57 http: TLS handshake error from 10.244.0.1:1572: EOF

The flagd sidecar in the demo-app pod has these logs (looks successful):

         ______   __       ________   _______    ______     /_____/\ /_/\     /_______/\ /______/\  /_____/\        \::::_\/_\:\ \    \::: _  \ \\::::__\/__\:::_ \ \        \:\/___/\\:\ \    \::(_)  \ \\:\ /____/\\:\ \ \ \        \:::._\/ \:\ \____\:: __  \ \\:\\_  _\/ \:\ \ \ \        \:\ \    \:\/___/\\:.\ \  \ \\:\_\ \ \  \:\/.:| |            \_\/     \_____\/ \__\/\__\/ \_____\/   \____/_/
2023-08-17T00:33:18.661Z    info    cmd/start.go:117    flagd version: v0.6.3 (bf74c5a3fc1b52108c2b057b3680576a90f31453), built at: 2023-08-04  {"component": "start"}2023-08-17T00:33:18.662Z  info    flag-evaluation/connect_service.go:199  metrics and probes listening at 8014    {"component": "service"}2023-08-17T00:33:18.662Z    info    kubernetes/kubernetes_sync.go:110   starting kubernetes sync notifier for resource: demo/featureflagconfiguration-sample    {"component": "sync", "sync": "kubernetes"}2023-08-17T00:33:18.663Z info    flag-evaluation/connect_service.go:179  Flag Evaluation listening at [::]:8030  {"component": "service"}2023-08-17T00:33:18.695Z    info    kubernetes/kubernetes_sync.go:218   kube sync notifier event: add: demo featureflagconfiguration-sample {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:34:18.699Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:35:18.699Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:36:18.700Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}2023-08-17T00:37:18.701Z info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:38:18.701Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:39:18.704Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:39:28.274Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:40:18.706Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:41:18.707Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:42:18.708Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:43:18.709Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:44:18.709Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:45:18.712Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:46:18.713Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:47:18.713Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:48:18.713Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:49:18.716Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:49:57.757Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:50:18.717Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:51:18.718Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:52:18.719Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:53:18.719Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
1023-08-17T00:54:18.720Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:55:18.720Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}
2023-08-17T00:56:18.723Z    info    kubernetes/kubernetes_sync.go:224   kube sync notifier event: update: demo featureflagconfiguration-sample  {"component": "sync", "sync": "kubernetes"}

The demo-app container itself has these logs:

Connecting to flagD at localhost:8030
Example app listening on port 3000
agardnerIT commented 1 year ago

Can verify that a curl to http://demoapp.demo from a pod within the demo namespace does produce a trace. But a request from a browser (via the port-forwarded browser) doesn't.

Why?

agardnerIT commented 1 year ago

image

Single request using port-forward means a single request to the backend - but no traces