Open ShangfengDing opened 6 months ago
can you do a hprof dump and check what's causing the mem consumption?
can you do a hprof dump and check what's causing the mem consumption?
From top, the RES section has exceeded the 1g memory set by the JVM
I open the Native Memory Tracking
jmap result:
I use the demo which provided by pyrocope also has memory leak
import io.pyroscope.http.Format;
import io.pyroscope.javaagent.PyroscopeAgent;
import io.pyroscope.javaagent.Snapshot;
import io.pyroscope.javaagent.api.Exporter;
import io.pyroscope.javaagent.api.Logger;
import io.pyroscope.javaagent.config.Config;
import io.pyroscope.javaagent.impl.DefaultConfigurationProvider;
import io.pyroscope.labels.Pyroscope;
import io.pyroscope.labels.LabelsSet;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class App {
public static final int N_THREADS = 8;
public static void main(String[] args) {
PyroscopeAgent.start(
new PyroscopeAgent.Options.Builder(
new Config.Builder()
.setApplicationName("demo.app{qweqwe=asdasd}")
.setServerAddress("http://localhost:4040")
.setFormat(Format.JFR)
.setLogLevel(Logger.Level.DEBUG)
.setLabels(mapOf("user", "tolyan"))
.build())
// .setExporter(new MyStdoutExporter())
.build()
);
Pyroscope.setStaticLabels(mapOf("region", "us-east-1"));
appLogic();
}
private static void appLogic() {
ExecutorService executors = Executors.newFixedThreadPool(N_THREADS);
for (int i = 0; i < N_THREADS; i++) {
executors.submit(() -> {
Pyroscope.LabelsWrapper.run(new LabelsSet("thread_name", Thread.currentThread().getName()), () -> {
while (true) {
try {
fib(32L);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
}
);
});
}
}
private static Map<String, String> mapOf(String... args) {
Map<String, String> staticLabels = new HashMap<>();
for (int i = 0; i < args.length; i += 2) {
staticLabels.put(args[i], args[i + 1]);
}
return staticLabels;
}
private static long fib(Long n) throws InterruptedException {
if (n == 0L) {
return 0L;
}
if (n == 1L) {
return 1L;
}
Thread.sleep(100);
return fib(n - 1) + fib(n - 2);
}
private static class MyStdoutExporter implements Exporter {
@Override
public void export(Snapshot snapshot) {
System.out.printf("Export %d %d%n", snapshot.data.length, snapshot.labels.toByteArray().length);
}
}
}
I use the 1g container to run this program, set -Xms256m -Xmx256m -XX:MaxDirectMemorySize=10m
the memory monitor:
Any updates on this one? I also faced with memory leak after adding Pyrosope agent
@dvovney did you check heap dump?
@korniltsev yes, I could not figure out what actually caused a leak. at some point we've added pyroscope into several services, and exactly after that all of them started leaking
I use 0.12.0 agent upload JFR data to my backend.
Upload code:
My exporter:
When I opened the agent report, I found that the memory was constantly rising, and I suspected there was a memory leak situation![image](https://github.com/grafana/pyroscope-java/assets/28734521/58bdceff-c78a-4977-86e4-1abc24a365ba)
Is there anyone who has encountered similar problems with me, or is there a problem with where I wrote it? Thanks