HTTP Status: 500 Internal Server Error

[Tiagodge]

Bug Report

I installed the open telemetry-demo on my OpenShift Cluster and I am receiving this 500 Server Error. Once I am not running locally I needed to create a load balancer to connect externally.

Browser error trying to access the page:

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.21.79.231:8080: connect: connection refused" could not retrieve cart main.(frontendServer).viewCartHandler /usr/src/app/handlers.go:280 main.instrumentHandler.func1 /usr/src/app/middleware.go:120 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2047 go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux.traceware.ServeHTTP /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux@v0.29.0/mux.go:145 github.com/gorilla/mux.(Router).ServeHTTP /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 main.(logHandler).ServeHTTP /usr/src/app/middleware.go:94 main.ensureSessionID.func1 /usr/src/app/middleware.go:148 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:2047 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2879 net/http.(conn).serve /usr/local/go/src/net/http/server.go:1930 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581

Symptom

I am having a CrashLoopBackOff error too: my-otel-demo-cart-service

1 Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false 2 Wasn't able to connect to redis 3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis 4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89 5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62 6 at cartservice.Startup.ConfigureServices(IServiceCollection services) in /usr/src/app/src/Startup.cs:line 38 7 at System.RuntimeMethodHandle.InvokeMethod(Object , Span`1& , Signature , Boolean , Boolean ) 8 at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo ) 9 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) 10 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>cDisplayClass9_0.gStartup|0(IServiceCollection serviceCollection) 11 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) 12 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>cDisplayClass8_0.b0(IServiceCollection services) 13 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance) 14 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.b__0(HostBuilderContext context, IServiceCollection services) 15 at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider() 16 at Microsoft.Extensions.Hosting.HostBuilder.Build() 17 at Program.

$(String[] args) in /usr/src/app/src/Program.cs:line 19

Redis is working well: 1 1:C 09 Aug 2022 13:40:02.649 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2 1:C 09 Aug 2022 13:40:02.657 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=1, just started 3 1:C 09 Aug 2022 13:40:02.658 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 4 1:M 09 Aug 2022 13:40:02.658 monotonic clock: POSIX clock_gettime 5 1:M 09 Aug 2022 13:40:02.663 Running mode=standalone, port=6379. 6 1:M 09 Aug 2022 13:40:02.664 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 7 1:M 09 Aug 2022 13:40:02.664 # Server initialized 8 1:M 09 Aug 2022 13:40:02.664 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 9 1:M 09 Aug 2022 13:40:02.665 * Ready to accept connections

[julianocosta89]

Hello @Tiagodge, could you try again with the new released version?

[Tiagodge]

Hello @julianocosta89 , I just tried and I continue to receive a crashloopbackoff on my-otel-demo-cart-service-855db6779c-ngrlt :

I removed the old one:

helm repo remove open-telemetry

And installed again:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install my-otel-demo open-telemetry/opentelemetry-demo

The error:

1 Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false 2 Wasn't able to connect to redis 3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis 4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89 5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62 6 at cartservice.Startup.ConfigureServices(IServiceCollection services) in /usr/src/app/src/Startup.cs:line 38 7 at System.RuntimeMethodHandle.InvokeMethod(Object , Span`1& , Signature , Boolean , Boolean ) 8 at System.Reflection.RuntimeMethodInfo.Invoke(Object , BindingFlags , Binder , Object[] , CultureInfo ) 9 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) 10 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>cDisplayClass9_0.gStartup|0(IServiceCollection serviceCollection) 11 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) 12 at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>cDisplayClass8_0.b0(IServiceCollection services) 13 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance) 14 at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.b__0(HostBuilderContext context, IServiceCollection services) 15 at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider() 16 at Microsoft.Extensions.Hosting.HostBuilder.Build() 17 at Program.

$(String[] args) in /usr/src/app/src/Program.cs:line 19

[fatsheep9146]

Could your check the status of the redis service?

[Tiagodge]

Yes @fatsheep9146 , Redis is running and opened for connections:

my-otel-demo-redis-844988fcfb-5m2pb

1 1:C 22 Aug 2022 18:05:40.990 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2 1:C 22 Aug 2022 18:05:40.990 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=1, just started 3 1:C 22 Aug 2022 18:05:40.990 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 4 1:M 22 Aug 2022 18:05:40.991 monotonic clock: POSIX clock_gettime 5 1:M 22 Aug 2022 18:05:40.992 Running mode=standalone, port=6379. 6 1:M 22 Aug 2022 18:05:40.993 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 7 1:M 22 Aug 2022 18:05:40.993 # Server initialized 8 1:M 22 Aug 2022 18:05:40.993 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 9 1:M 22 Aug 2022 18:05:40.993 * Ready to accept connections

[julianocosta89]

@Tiagodge is this issue still ongoing? We had some new updates in the demo and in the Helm charts, would you mind taking a look again?

[Tiagodge]

Yes, the issue was not fixed, sure. I will try the new one. But I am on vacation returning on oct 23

Thanks

[austinlparker]

This is non-blocking for 1.0

[lfranchini31]

@Tiagodge you have another services in CrashLoopBackOff error? I try the last version of this helm chart and i have 4 pods in error (my-otel-demo-cart-service with the same error, my-otel-demo-email-service, my-otel-demo-featureflag-service and my-otel-demo-frontend always in CrashLoopBackOff). I think these errors have the same source: [error] Could not create schema migrations table. This error usually happens due to the following:

• The database does not exist
• The "schema_migrations" table, which Ecto uses for managing migrations, was defined by another library
• There is a deadlock while migrating (such as using concurrent indexes with a migration_lock) To fix the first issue, run "mix ecto.create". To address the second, you can run "mix ecto.drop" followed by "mix ecto.create". Alternatively you may configure Ecto to use another table and/or repository for managing migrations: config :featureflagservice, Featureflagservice.Repo, migration_source: "some_other_table_for_schema_migrations", migration_repo: AnotherRepoForSchemaMigrations

** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2977ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval See DBConnection.start_link/2 for more information (ecto_sql 3.8.2) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1 (elixir 1.13.3) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2 (ecto_sql 3.8.2) lib/ecto/adapters/sql.ex:1024: Ecto.Adapters.SQL.execute_ddl/4 (ecto_sql 3.8.2) lib/ecto/migrator.ex:696: Ecto.Migrator.verbose_schema_migration/3 (ecto_sql 3.8.2) lib/ecto/migrator.ex:510: Ecto.Migrator.lock_for_migrations/4 (ecto_sql 3.8.2) lib/ecto/migrator.ex:422: Ecto.Migrator.run/4 (ecto_sql 3.8.2) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3 (featureflagservice 0.1.0) lib/featureflagservice/release.ex:12: anonymous fn/2 in Featureflagservice.Release.migrate/0

[cartersocha]

Is this an issue with the current version @lfranchini31, @Tiagodge

[lfranchini31]

@cartersocha Yes, i think so :)

[austinlparker]

If this issue reoccurs with the current released version, please re-open this issue.

[Tiagodge]

I just tried again and the problem persists

I cleaned up everything in the cluster and apply the new version, but now, with the new version I can navigate on the application, open Jaeger and everything, but is failing when I try to add items to the cart.

the error: Connecting to Redis: my-otel-demo-redis:6379,ssl=false,allowAdmin=true,abortConnect=false 2 Wasn't able to connect to redis 3 Unhandled exception. System.ApplicationException: Wasn't able to connect to redis 4 at cartservice.cartstore.RedisCartStore.EnsureRedisConnected() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 89 5 at cartservice.cartstore.RedisCartStore.InitializeAsync() in /usr/src/app/src/cartstore/RedisCartStore.cs:line 62 6 at Program.

$(String[] args) in /usr/src/app/src/Program.cs:line 40

Redis is accepting connection:

1:C 07 Feb 2023 18:34:44.280 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2 1:C 07 Feb 2023 18:34:44.280 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=1, just started 3 1:C 07 Feb 2023 18:34:44.280 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf 4 1:M 07 Feb 2023 18:34:44.286 monotonic clock: POSIX clock_gettime 5 1:M 07 Feb 2023 18:34:44.291 Running mode=standalone, port=6379. 6 1:M 07 Feb 2023 18:34:44.292 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 7 1:M 07 Feb 2023 18:34:44.292 # Server initialized 8 1:M 07 Feb 2023 18:34:44.292 # WARNING Your system is configured to use the 'xen' clocksource which might lead to degraded performance. Check the result of the [slow-clocksource] system check: run 'redis-server --check-system' to check if the system's clocksource isn't degrading performance. 9 1:M 07 Feb 2023 18:34:44.295 * Ready to accept connections