We have a customer which is seeing their gateway-proxy memory increased when making any change on any configuration, for example modifying a VirtualService. When they make a modification, they see its memory increasing and not releasing, for example going from 8GB to 11GB by just making a change.
They have spikes up to 10.8 GB on the envoy_server_memory_physical_size, 11.5 GB on the envoy_server_memory_heap_size and 10.2 GB on the POD level.
I have captured one of their heaps before making the modification (envoy.heap) and after the modification envoy-after.heap, in heaps.zip file
And they just see these errors, but I would say it is more related to modsec:
{"level":"error","ts":"2023-11-13T16:04:08.454Z","logger":"gloo-ee.v1.event_loop.setup.v1.event_loop.syncer.kubernetes_eds","caller":"kubernetes/eds.go:208","msg":"upstream gloo: port 8080 not found for service upstream-svc","version":"1.14.0","stacktrace":"github.com/solo-io/gloo/projects/gloo/pkg/plugins/kubernetes.(*edsWatcher).List\n\t/go/pkg/mod/github.com/solo-io/gloo@v1.14.0/projects/gloo/pkg/plugins/kubernetes/eds.go:208\ngithub.com/solo-io/gloo/projects/gloo/pkg/plugins/kubernetes.(*edsWatcher).watch.func1\n\t/go/pkg/mod/github.com/solo-io/gloo@v1.14.0/projects/gloo/pkg/plugins/kubernetes/eds.go:229\ngithub.com/solo-io/gloo/projects/gloo/pkg/plugins/kubernetes.(*edsWatcher).watch.func2\n\t/go/pkg/mod/github.com/solo-io/gloo@v1.14.0/projects/gloo/pkg/plugins/kubernetes/eds.go:256"}
{"level":"error","ts":1699885178.0106442,"logger":"rate-limiter","msg":"finished unary call with code Unknown","version":"undefined","grpc.start_time":"2023-11-13T14:19:38Z","grpc.request.deadline":"2023-11-13T14:19:38Z","system":"grpc","span.kind":"server","grpc.service":"envoy.service.ratelimit.v3.RateLimitService","grpc.method":"ShouldRateLimit","error":"Could not execute Redis pipeline: context canceled","errorVerbose":"Could not execute Redis pipeline\n\tgrpc.(*Server).handleStream:/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704\n\tgrpc.(*Server).processUnaryRPC:/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336\n\tv3._RateLimitService_ShouldRateLimit_Handler:/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.11.0/envoy/service/ratelimit/v3/rls.pb.go:996\n\tgo-grpc-middleware.ChainUnaryServer.func1:/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\n\tgo-grpc-middleware.ChainUnaryServer.func1.1.1:/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n\tzap.UnaryServerInterceptor.func1:/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/logging/zap/server_interceptors.go:31\n\tgo-grpc-middleware.ChainUnaryServer.func1.1.1:/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n\thealthchecker.GrpcUnaryServerHealthCheckerInterceptor.func1:/go/pkg/mod/github.com/solo-io/go-utils@v0.24.0/healthchecker/grpc.go:69\n\tv3._RateLimitService_ShouldRateLimit_Handler.func1:/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.11.0/envoy/service/ratelimit/v3/rls.pb.go:994\n\tservice.(*service).ShouldRateLimit:/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/service/ratelimit.go:223\n\tservice.(*service).shouldRateLimitWorker:/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/service/ratelimit.go:171\n\tredis.(*rateLimitCacheImpl).DoLimit:/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/cache/redis/cache_impl.go:149\n\tredis.pipelineFetch:/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/cache/redis/cache_impl.go:48\n\tredis.(*connectionImpl).Pop:/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/cache/redis/driver_impl.go:146\ncontext canceled","grpc.code":"Unknown","grpc.time_ms":0.445,"stacktrace":"github.com/grpc-ecosystem/go-grpc-middleware/logging/zap.DefaultMessageProducer\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/logging/zap/options.go:212\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/logging/zap/server_interceptors.go:39\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/envoyproxy/go-control-plane/envoy/service/ratelimit/v3._RateLimitService_ShouldRateLimit_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.11.0/envoy/service/ratelimit/v3/rls.pb.go:996\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965"}
{"level":"dpanic","ts":1699885303.8779607,"logger":"rate-limiter","msg":"Could not execute Redis pipeline: context canceled","version":"undefined","stacktrace":"github.com/solo-io/rate-limiter/pkg/cache/redis.(*connectionImpl).Pop\n\t/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/cache/redis/driver_impl.go:145\ngithub.com/solo-io/rate-limiter/pkg/cache/redis.pipelineFetch\n\t/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/cache/redis/cache_impl.go:48\ngithub.com/solo-io/rate-limiter/pkg/cache/redis.(*rateLimitCacheImpl).DoLimit\n\t/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/cache/redis/cache_impl.go:149\ngithub.com/solo-io/rate-limiter/pkg/service.(*service).shouldRateLimitWorker\n\t/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/service/ratelimit.go:171\ngithub.com/solo-io/rate-limiter/pkg/service.(*service).ShouldRateLimit\n\t/go/pkg/mod/github.com/solo-io/rate-limiter@v0.8.0/pkg/service/ratelimit.go:223\ngithub.com/envoyproxy/go-control-plane/envoy/service/ratelimit/v3._RateLimitService_ShouldRateLimit_Handler.func1\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.11.0/envoy/service/ratelimit/v3/rls.pb.go:994\ngithub.com/solo-io/go-utils/healthchecker.GrpcUnaryServerHealthCheckerInterceptor.func1\n\t/go/pkg/mod/github.com/solo-io/go-utils@v0.24.0/healthchecker/grpc.go:69\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/logging/zap.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/logging/zap/server_interceptors.go:31\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/envoyproxy/go-control-plane/envoy/service/ratelimit/v3._RateLimitService_ShouldRateLimit_Handler\n\t/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.11.0/envoy/service/ratelimit/v3/rls.pb.go:996\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965"}
They are running GE 1.14.0, I guess if every time we make a modification in a VS, all waf rules from every VS are loaded again in the gateway-proxy, or just asking myself how can be this possible, as memory grows when they change anything, but we have validation disabled
They also have discovery disabled:
gloo-ee:
gloo:
discovery:
enabled: false
And
disableKubernetesDestinations: true
Expected Behavior
Making config changes shouldn't cause this memory growths, maybe it is waf rules that are reloaded every time the config changes
This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.
Gloo Edge Product
Enterprise
Gloo Edge Version
1.14.0
Kubernetes Version
1.25
Describe the bug
We have a customer which is seeing their gateway-proxy memory increased when making any change on any configuration, for example modifying a VirtualService. When they make a modification, they see its memory increasing and not releasing, for example going from 8GB to 11GB by just making a change. They have spikes up to 10.8 GB on the envoy_server_memory_physical_size, 11.5 GB on the envoy_server_memory_heap_size and 10.2 GB on the POD level.
I have captured one of their heaps before making the modification (envoy.heap) and after the modification envoy-after.heap, in heaps.zip file
heaps.zip
It is worth to say that they are using was in heavily in most of their VirtualServices:
And they have validation disabled at gateway-proxy level, you can check settings.yaml:
When they make any kind of change, waf rules are logged in the gateway-proxy:
And they just see these errors, but I would say it is more related to modsec:
They are running GE 1.14.0, I guess if every time we make a modification in a VS, all waf rules from every VS are loaded again in the gateway-proxy, or just asking myself how can be this possible, as memory grows when they change anything, but we have validation disabled
They also have discovery disabled:
And
Expected Behavior
Making config changes shouldn't cause this memory growths, maybe it is waf rules that are reloaded every time the config changes
Additional Environment Detail
No response
Additional Context
No response