search

uber-common / jvm-profiler

JVM Profiler Sending Metrics to Kafka, Console Output or Custom Reporter
Other
1.78k stars 342 forks source link

Cross Node Communication #44

Closed rawlani closed 3 years ago

rawlani commented 5 years ago

Hi, I have been using Uber JVM profiler for profiling my Spark applications. May I know if the profiler provides information on cross node communication?

hiboyang commented 5 years ago

Hi, glad you are using Uber JVM profiler! What do you mean "cross node communication"?

rawlani commented 5 years ago

Hi, apologies for the delay. By Cross-node communication, I mean the Information on the data that is written and read between the multiple worker nodes and between the driver nodes and worker nodes.

hiboyang commented 5 years ago

I see. Thanks for explaining. It does not support such information then.

rawlani commented 5 years ago

Okay. Thanks for letting us know. Moreover, I also wanted to know if we can pull up any information about I/O stuff on each node?

hiboyang commented 5 years ago

There is some I/O stuff you could get, but not comprehensive IO metrics. You could add "ioProfiling=true" in the java agent parameters. And you could find what IO metrics it collect from this source file: https://github.com/uber-common/jvm-profiler/blob/master/src/main/java/com/uber/profiling/profilers/IOProfiler.java

hiboyang commented 3 years ago

Close inactive issue.