search

open-telemetry / opentelemetry-demo

This repository contains the OpenTelemetry Astronomy Shop, a microservice-based distributed system intended to illustrate the implementation of OpenTelemetry in a near real-world environment.
https://opentelemetry.io/docs/demo/
Apache License 2.0
1.73k stars 1.09k forks source link

CrashLoopBackOff on the my-otel-demo-cartservice running on OpenShift #736

Closed Tiagodge closed 1 year ago

Tiagodge commented 1 year ago

Bug Report

I am using the version 730: https://github.com/open-telemetry/opentelemetry-demo/pull/730

Symptom

The cart service is not working, I can access the website, but cannot add items to the cart

I cleaned up everything in the cluster and apply the new version, but now, with the new version I can navigate on the application, open Jaeger and everything, but is failing when I try to add items to the cart.

the error: Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false 2 Wasn't able to connect to redis 3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis 4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89 5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62 6 at Program.

$(String[] args) in /usr/src/app/src/Program.cs:line 40

But as you can see here, Redis is accepting connection:

1:C 07 Feb 2023 18:34:44.280 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2 1:C 07 Feb 2023 18:34:44.280 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started 3 1:C 07 Feb 2023 18:34:44.280 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 4 1:M 07 Feb 2023 18:34:44.286 monotonic clock: POSIX clock_gettime 5 1:M 07 Feb 2023 18:34:44.291 Running mode=standalone, port=6379. 6 1:M 07 Feb 2023 18:34:44.292 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 7 1:M 07 Feb 2023 18:34:44.292 # Server initialized 8 1:M 07 Feb 2023 18:34:44.292 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 9 1:M 07 Feb 2023 18:34:44.295 * Ready to accept connections

julianocosta89 commented 1 year ago

@Tiagodge thanks for reporting this. How are you deploying the sample? (Docker or K8s) Is the redis container running? Which version of the demo are you using? (you have referred to 730, but this is from the docs, and not from the demo)

Tiagodge commented 1 year ago

I deployed via helm chart using exactly this command:

helm install my-otel-demo open-telemetry/opentelemetry-demo

Yes, Redis pod is running and accepting connections as we can see on the pod log:

9 1:M 07 Feb 2023 18:34:44.295 * Ready to accept connections

This is the version I am using:

apiVersion: v2 appVersion: 1.3.0 dependencies:

julianocosta89 commented 1 year ago

@puckpuck and @TylerHelmuth any ideas here?

The Cart pod is in CrashLoopBackOff and can't seem to be able to connect to Redis, even though Redis is up and running.

puckpuck commented 1 year ago

These two warnings are intriguing. I don't see them on my side in any of the deployments I have running (also using Redis 7.0.10 now instead of 7.0.8)

6 1:M 07 Feb 2023 18:34:44.292 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
7 1:M 07 Feb 2023 18:34:44.292 # Server initialized
8 1:M 07 Feb 2023 18:34:44.292 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to

Can you try reinstalling again? Redis is up to 7.0.10 now, and though I doubt that would be the cause, I want to ensure we are all using the same version.

puckpuck commented 1 year ago

The Helm chart has been updated to work with OpenShift. There are a few special instructions you need to follow in the Helm chart's readme for OpenShift installation.