temporalio / temporal

Temporal service
https://docs.temporal.io
MIT License
10.57k stars 769 forks source link

503 in api/v1/namespaces/<your_namespace>/search-attributes? sometime #5436

Open granescb opened 4 months ago

granescb commented 4 months ago

Expected Behavior

200 http status

Actual Behavior

http 503 sometime image

Steps to Reproduce the Problem

  1. Open "api/v1/namespaces//search-attributes?" in browser
  2. Open browser console
  3. Update page 20-30 times to get some errors

Specifications

The error from the component temporal-frontend: {"level":"error","ts":"2024-02-19T13:17:16.265Z","msg":"service failures","operation":"OperatorListSearchAttributes","wf-namespace":"les","error":"Unable to get namespace info with error: les","logging-call-at":"telemetry.go:341","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:341\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:174\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:180\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/builder/temporal/common/authorization/interceptor.go:158\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:111\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1195\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1186\ngo.temporal.io/api/operatorservice/v1._OperatorService_ListSearchAttributes_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.24.1-0.20231003165936-bb03061759c8/operatorservice/v1/service.pb.go:302\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1376\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:1753\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.2/server.go:998"}

granescb commented 4 months ago

Probably, the error is rising here: https://github.com/temporalio/temporal/blob/b670c79114d2dacd07bb53156281abe10e96473e/service/frontend/operator_handler.go#L518-L526 And there is no information about which exactly error happens in the logs, only namespace name.

granescb commented 4 months ago

Fixed typo here https://github.com/temporalio/temporal/pull/5473