Significant memory usage

RobicToNieMaKomu commented 9 months ago

Integration

sentry-spring-boot-starter-jakarta

Java Version

17

Version

7.1.0

Steps to Reproduce

Our service is running on SB 3.1.8 & reactive stack (Webflux).

Baseline. Without Sentry, memory consumption oscillates around 700 Mb (committed ~900 Mb, and max is 1.88 Gb).

After enabling integration on the defaults (with tracing disabled), mem usage went up to the ceiling

sentry:
dsn: ...
enabled: false
enableTracing: false
environment: dev
serverName: server
shutdownTimeout: 0

After further tuning, I was able to reduce the memory:

  SENTRY_MAXBREADCRUMBS: 1
  SENTRY_ENABLEUSERINTERACTIONBREADCRUMBS: false
  SENTRY_DIAGNOSTICLEVEL: fatal
  SENTRY_MAXCACHEITEMS: 0
  SENTRY_ATTACHSTACKTRACE: false

However, the situation is still unacceptable, as integration increased RAM usage by 50% (!). Can we do something to make it better? Can breadcrumbs be entirely disabled? They are useless in our case as they are not correlated with the request causing an actual error.

Expected Result

Sane memory/CPU footprint.

Actual Result

Gigantic memory consumption.

adinauer commented 9 months ago

@RobicToNieMaKomu thanks for opening this issue. We'll take a look and update here.

adinauer commented 9 months ago

@RobicToNieMaKomu I can confirm, we're using up more memory on startup but after manually triggering GC, it ends up roughly the same.

How I tested

I fired up our Spring Boot 3 Webflux sample
let it startup for a bit then manually triggered GC in visualvm a couple times
sent some requests in using wrk -t2 -c4 -d30s -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" http://127.0.0.1:8080/todo-webclient/1. This sends ~ 4k requests in 30s:

Running 30s test @ http://127.0.0.1:8080/todo-webclient/1
  2 threads and 4 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.21ms    9.48ms 174.51ms   93.47%
    Req/Sec    72.24     11.44   101.00     68.91%
  4325 requests in 30.03s, 532.18KB read
Requests/sec:    144.01
Transfer/sec:     17.72KB

manually triggered GC a couple times again
reran wrk
manually triggered GC a couple times again

The endpoint does not record any errors but did create transactions and spans (when Sentry was enabled).

Results

Here's memory with Sentry:

And memory without Sentry:

So with Sentry enabled the server is consuming a lot more memory on startup but it is able to free that back up. Will keep investigating.

Questions

Can you please tell us a bit about your usage of Sentry on the screenshots you provided?

Do you do any manual capturing of errors, breadcrumbs etc.?
Other interaction with the Sentry API?
Are any/many exceptions thrown and recorded by Sentry?
Roughly how many requests is the server seeing?
Do you use CRONS? There's another potential memory problem there, see https://github.com/getsentry/sentry-java/issues/3165

adinauer commented 9 months ago

@RobicToNieMaKomu if you're not using any of our AOP features (@SentryTransaction, @SentrySpan, @SentryCheckIn annotations) you could try to disable those using sentry.enable-aot-compatibility=true and see if that helps lower the initial memory consumption. From my testing it only used ~ 1/3 of memory on startup this way.

adinauer commented 9 months ago

Here's another screenshot showing memory consumption with above mentioned test procedure and sentry.enable-aot-compatibility=true:

RobicToNieMaKomu commented 9 months ago

Hey @adinauer , thank's for looking into that!

Do you do any manual capturing of errors, breadcrumbs etc.?

We don't create/capture/use breadcrumbs. We catch exceptions above the controller's layer in the @ControllerAdvice class -> @ExceptionHandler methods.

Other interaction with the Sentry API?

Nope.

Are any/many exceptions thrown and recorded by Sentry?

Not many, during a few days, we got <100 occurrences of 3 different Exceptions

Roughly how many requests is the server seeing?

Single pod/instance handles roughly up to 150 RPS

Do you use CRONS?

Nope, at least not directly in our code (not sure if it's used inside Spring itself)

I just set sentry.enable-aot-compatibility=true, will report back tomorrow with the result.

adinauer commented 9 months ago

@RobicToNieMaKomu you can also try playing with these settings:

sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

and you could drop all breadcrumbs from BeforeBreadcrumbCallback. Should be as simple as providing a spring bean that always returns null from Breadcrumb execute(@NotNull Breadcrumb breadcrumb, @NotNull Hint hint).

RobicToNieMaKomu commented 9 months ago

This is what it looks like after applying these properties:

sentry.enable-aot-compatibility=true
sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

and BeforeBreadcrumbCallback -> null:

Maybe memory consumption became slightly smaller, but it still has a significant impact.

adinauer commented 9 months ago

@RobicToNieMaKomu can you please provide a sample that allows us to reprodue the issue, maybe by forking the repo and adding to our Spring Boot 3 Webflux Sample. I'm assuming you're using additional Spring features, settings or other libraries that could cause this when combined with Sentry.

and BeforeBreadcrumbCallback -> null

I'm not sure we're on the same page here. Did you provide a callback that returns null (this is what is needed to drop all breadcrumbs) or did you set the callback to null (this is the default behaviour of not dropping breadcrumbs)?

sentry.logging.minimum-breadcrumb-level=debug

I meant to suggest you can increase the level to not have as many breadcrumbs but it's not that relevant if you're dropping all breadcrumbs anyways or limit breadcrumb count to 1.

adinauer commented 9 months ago

Oh and are you using any special garbage collector settings?

markushi commented 8 months ago

Hey @RobicToNieMaKomu, did you had some time to look into the questions from above^, by any chance?

RobicToNieMaKomu commented 8 months ago

@adinauer I'm sorry, I can't fork the repo (NDA) and don't have enough time to come up with PoC now. The simplified stack we're using in that service:

        <java.version>17</java.version>
        <lombok.version>1.18.30</lombok.version>
        <springboot-cloud.version>2022.0.5</springboot-cloud.version>
        <resilience4j.version>2.1.0</resilience4j.version>
        <protobuf-java.version>3.22.2</protobuf-java.version>
        <awaitility.version>4.2.0</awaitility.version>

Besides, it heavily uses Redis (spring-boot-starter-data-redis-reactive dependency of SB 3.1.8).

I'm not sure we're on the same page here. Did you provide a callback that returns null (this is what is needed to drop all breadcrumbs) or did you set the callback to null (this is the default behaviour of not dropping breadcrumbs)?

I provided a callback that returns null.

are you using any special garbage collector settings?

Java options: -Xss512k -XX:+UseZGC -XX:MaxDirectMemorySize=64m -XX:ReservedCodeCacheSize=120M

adinauer commented 8 months ago

Thanks for the update.

getsentry / sentry-java