search

concourse / hush-house

Concourse k8s-based environment
https://hush-house.pivotal.io
29 stars 23 forks source link

jaeger: increaese retention and use a pesistent storage type #119

Closed cirocosta closed 4 years ago

cirocosta commented 4 years ago

Hey,

At the moment, jaeger is configured to use the memory-based storage type, with (I guess) the default retention limit, which seems to be growing more than the machines we use can really hld:

NAME                                      READY   STATUS    RESTARTS   AGE
jaeger-jaeger-operator-5878c646f5-fjvm7   1/1     Running   0          7d2h
simplest-6fbbdc7794-9sc8p                 0/1     Evicted   0          5d
simplest-6fbbdc7794-mckk6                 1/1     Running   0          4h13m
simplest-6fbbdc7794-ptz2w                 0/1     Evicted   0          5d6h
simplest-6fbbdc7794-vdbfp                 0/1     Evicted   0          5d14h
simplest-6fbbdc7794-w7cs7                 0/1     Evicted   0          7d6h
simplest-6fbbdc7794-zxx25                 0/1     Evicted   0          17h

...
Message:        The node was low on resource: memory. Container jaeger was using 10083764Ki, which exceeds its request of 0.

It seems like a simple-enough approach would be to have the badger storage type enabled, without the need of bringing an entire "full on" storage backend (cassandra/elasticsearch) up: https://www.jaegertracing.io/docs/1.13/deployment/#badger-local-storage

Aside from that, it'd be good to have the pods deployed with proper requests and resource limits, rather than consume those on a best-effort basis.

Thanks