Reference for fields used

[ajames-branch]

Could you please provide some material/ links that explains what each field means in the profiler output.

I am able to get profiler output for my spark streaming application, but finding it hard to understand the fields and its meaning.

I am specifically interested in CpuAndMemory and MethodDuration output fields

[ajames-branch]

@boy-uber @felixcheung highly appreciate and thankful if anyone can help me here.

[hiboyang]

@ajames-branch For CpuAndMemory profiler, you could check its source code. It should be straightforward to understand each field, like following code:

        map.put("processCpuLoad", processCpuLoad);
        map.put("systemCpuLoad", systemCpuLoad);
        map.put("processCpuTime", processCpuTime);

        map.put("heapMemoryTotalUsed", heapMemoryTotalUsed);
        map.put("heapMemoryCommitted", heapMemoryCommitted);

        map.put("nonHeapMemoryTotalUsed", nonHeapMemoryTotalUsed);
        map.put("nonHeapMemoryCommitted", nonHeapMemoryCommitted);

For MethodDuration, it capture each method's duration and aggregate the numbers by each minute. For example, if you profile a method foo.ClassA.methodBar, MethodDuration profiler will generate four records each minute. Each record will contain following fields: className, methodName, metricName, metricValue. Following are examples of the four records:

className	methodName	metricName	metricValue
foo.ClassA	methodBar	duration.count	3 (how many times this method is called during that	minute)
foo.ClassA	methodBar	duration.sum	30 (sum of all the durations of this method call)
foo.ClassA	methodBar	duration.min	5 (min value of all the durations of this method call)
foo.ClassA	methodBar	duration.max	10 (max value of all the durations of this method call)

The explanation for values in the upper is: duration.count: that method is called 3 times during that minute. duration.sum: the total time spends on that method is 30 milliseconds. duration.min: the shortest run of this method takes 5 milliseconds duration.min: the longest run of this method takes 10 milliseconds

[ajames-branch]

Thank You @boy-uber This was really helpful :-)