k8s deployment issues

[fcrisciani]

Requirement - what kind of business use case are you trying to solve?

deploy jeager

Problem - what in Jaeger blocks you from solving the requirement?

I tried both the all-in-one as well as the production yaml. To verify basic health of the service was trying to open the UI but looks like the container is not listening on the port 16686. I can see a log like:

{"level":"info","ts":1532641841.5165153,"caller":"standalone/main.go:304","msg":"Starting jaeger-query HTTP server","port":0}

does the all-in-one expect some configuration?

This is actually the full log of the pod:

{"level":"info","ts":1532641841.3107834,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1532641841.3112895,"caller":"memory/factory.go:55","msg":"Memory storage configuration","configuration":{"MaxTraces":0}}
{"level":"info","ts":1532641841.3155985,"caller":"tchannel/builder.go:94","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1532641841.3156722,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"127.0.0.1:14267"}
{"level":"info","ts":1532641841.3163402,"caller":"standalone/main.go:187","msg":"Starting agent"}
{"level":"info","ts":1532641841.3170893,"caller":"standalone/main.go:227","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1532641841.3171778,"caller":"standalone/main.go:237","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1532641841.3713417,"caller":"standalone/main.go:257","msg":"Listening for Zipkin HTTP traffic","zipkin.http-port":9411}
{"level":"info","ts":1532641841.51646,"caller":"standalone/main.go:298","msg":"Registering metrics handler with jaeger-query HTTP server","route":"/metrics"}
{"level":"info","ts":1532641841.5165153,"caller":"standalone/main.go:304","msg":"Starting jaeger-query HTTP server","port":0}
{"level":"info","ts":1532641841.516577,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1532641842.3158925,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1532641842.3159378,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"127.0.0.1:14267"}
{"level":"info","ts":1532641842.3163357,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}

Also this one does not look too good:

{"level":"info","ts":1532641842.3158925,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1}

do you have any suggestion on how to debug this?

[yurishkuro]

You can ignore the last one, it's normal (although we could improve the logging not to log that on initial connect).

The port:0 for query service is the main issue. Normally this port is defaulted to 16686, you must have something that overrides it via cli flag or env var or config. Maybe you're missing some variable substitution.

[fcrisciani]

@yurishkuro I just deployed like in the docs: kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml with the only modification to use NodePort instead of Loadbalancer for the service.

And the describe for the deployment does not mention any env variable a part from COLLECTOR_ZIPKIN_HTTP_PORT:

Name:               jaeger-deployment
Namespace:          test
CreationTimestamp:  Thu, 26 Jul 2018 14:50:38 -0700
Labels:             app=jaeger
                    jaeger-infra=jaeger-deployment
Annotations:        deployment.kubernetes.io/revision=1
Selector:           app=jaeger,jaeger-infra=jaeger-pod
Replicas:           1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:       Recreate
MinReadySeconds:    0
Pod Template:
  Labels:  app=jaeger
           jaeger-infra=jaeger-pod
  Containers:
   jaeger:
    Image:      jaegertracing/all-in-one
    Ports:      5775/UDP, 6831/UDP, 6832/UDP, 5778/TCP, 16686/TCP, 9411/TCP
    Readiness:  http-get http://:14269/ delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      COLLECTOR_ZIPKIN_HTTP_PORT:  9411
    Mounts:                        <none>
  Volumes:                         <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   jaeger-deployment-b8bfddc7b (1/1 replicas created)
Events:          <none>

Is there something that has to be actually specified? Can I forcefully specify the query port for the all-in-one?

[fcrisciani]

@yurishkuro I was actually lying let me get back to you, I checked the runtime and there is a huge list of env variables related to other services.

[fcrisciani]

ok deploying actually in a different namespace did the trick, going to close this one and will start investigating which env created the problem. Thanks for the help

[fcrisciani]

Just to give closure, I solved specifying the port as a command argument directly in the yaml file" --query.port=<port>