OutOfMemoryError in SentryTracer

[romtsn]

Description

From play console:

Exception java.lang.OutOfMemoryError:
  at java.lang.Thread.nativeCreate
  at java.lang.Thread.start (Thread.java:883)
  at java.util.Timer.<init> (Timer.java:183)
  at java.util.Timer.<init> (Timer.java:153)
  at io.sentry.SentryTracer.<init> (SentryTracer.java:105)
  at io.sentry.Hub.createTransaction (Hub.java:727)
  at io.sentry.Hub.startTransaction (Hub.java:677)
  at io.sentry.Sentry.startTransaction (Sentry.java:892)
  at io.sentry.HubAdapter.startTransaction (HubAdapter.java:202)
  at io.sentry.android.core.ActivityLifecycleIntegration.startTracing (ActivityLifecycleIntegration.java:227)
  at io.sentry.android.core.ActivityLifecycleIntegration.onActivityCreated (ActivityLifecycleIntegration.java:358)
  at android.app.Application.dispatchActivityCreated (Application.java:379)
  at android.app.Activity.dispatchActivityCreated (Activity.java:1263)
  at android.app.Activity.onCreate (Activity.java:1536)
  at androidx.core.app.ComponentActivity.onCreate (ComponentActivity.java:88)
  at androidx.activity.ComponentActivity.onCreate (ComponentActivity.java:378)
  at androidx.fragment.app.FragmentActivity.onCreate (FragmentActivity.java:217)
  at <private>
  at android.app.Activity.performCreate (Activity.java:7866)
  at android.app.Activity.performCreate (Activity.java:7855)
  at android.app.Instrumentation.callActivityOnCreate (Instrumentation.java:1306)
  at android.app.ActivityThread.performLaunchActivity (ActivityThread.java:3273)
  at android.app.ActivityThread.handleLaunchActivity (ActivityThread.java:3437)
  at android.app.servertransaction.LaunchActivityItem.execute (LaunchActivityItem.java:83)
  at android.app.servertransaction.TransactionExecutor.executeCallbacks (TransactionExecutor.java:135)
  at android.app.servertransaction.TransactionExecutor.execute (TransactionExecutor.java:95)
  at android.app.ActivityThread$H.handleMessage (ActivityThread.java:2041)
  at android.os.Handler.dispatchMessage (Handler.java:107)
  at android.os.Looper.loop (Looper.java:214)
  at android.app.ActivityThread.main (ActivityThread.java:7386)
  at java.lang.reflect.Method.invoke
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:492)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:982)

[adinauer]

Is there a new spike of these or could they just be coming from devices with very limited resources? In other words should we investigate any newly introduced increase in memory consumption?

[markushi]

Let's start simple:

• Integrate our SDK into an OSS app
• check number of threads / memory consumed by SDK

[markushi]

A heap analysis done by a customer shows Span objects holding large stacktraces (preventing GC) for a timeframe of over 2+ hours. This definitely doesn't sound right.

[markushi]

After a quick test with our sentry-springboot example I can't seem to reproduce this easily. Scenario:

• Use sentry-springboot example

• Capture an exception on every request

  @PostMapping
  Person create(@RequestBody Person person) {
  Sentry.captureException(new IllegalStateException("oh no"));
  return personService.create(person);
  }

• Flood the service with 1k requests

  i=0
  max=1000
  while [ $i -lt $max ]
  do
  curl -XPOST --user user:password http://localhost:8080/person/ -H "Content-Type:application/json" -d '{"firstName":"John","lastName":"Smith"}'
  true $(( i++ ))
  sleep 2
  done

• Analyze the heap using mat

  

[markushi]

Tried a few variations as well, same picture: no leaking spans.

1. Throwing an uncaught exception

   @PostMapping
   Person create(@RequestBody Person person) {
   throw new IllegalStateException("oh no");
   // return personService.create(person);
   }

2. Attaching a stack trace to a span

   @PostMapping
   Person create(@RequestBody Person person) {
   final ISpan span = Sentry.getSpan().startChild("op.bad");
   span.setThrowable(new RuntimeException("bad things happened"));
   span.finish(SpanStatus.UNKNOWN_ERROR);
   
   return personService.create(person);
   }

[markushi]

@lbloder we recently got a report from a sentry-java springboot customer regarding spans with stacktraces leaking and leading to OOMs. I tried a few basics (See comments ^ above). Feel free to chime in if you have any more ideas what could cause spans to be retained.

Some more details from our customer setup:

• They do not have any integrations active
• They don’t use any extra manual instrumentation
• They utilise Sentry across their spring-boot microservices which are all hosted on AKS servers
• These services all communicate via REST APIs and they receive calls from a browser-based UI.
• They noticed the usage by performing a heap-dump analysis.
• They note “Sentry spans hold the exception object and blocks the garbage collection of these exception objects even after 2+ hours.”

I'm wondering if this could be related to some background workers having long running transactions.

[lbloder]

@markushi I'm a bit confused on They do not have any integrations active:

• How are they using Sentry?
  • My guess is, they are adding the sentry-spring-boot dependency manually, is that correct?
  • Or are they using the gradle or maven plugin for installation?
  • Or are they using the vanilla sentry-java sdk?

Also, it might be of relevance if they are using SpringBoot < 3 or not. And, one more thing that could help us narrow down the search space is whether they are using WebFlux.

I'll have a look into that as well.

[markushi]

@lbloder thanks for the details, let me reach out to them!

[stefanosiano]

it occurred also in CpuCollector on Android