Ref

[at15]

Existing benchmark result and benchmark tools

• Popper has spark perf but I don't think I can use it directly
• https://www.oreilly.com/ideas/investigating-sparks-performance mentioned CPU is the bottleneck but the webcast is no longer available

Benchmark frameworks

• https://github.com/intel-hadoop/HiBench support both Hadoop and Spark

Perf

• perf for monitor CPU and find 'area of interest'
• Java may need to use map, see 4.3. JIT Symbols (Java, Node.js) section
• https://github.com/jrudolph/perf-map-agent
• perf seems to have probe for java now https://github.com/torvalds/linux/tree/master/tools/perf/jvmti
• https://github.com/brendangregg/perf-tools a collection of shell scripts but focus on kernel 3.2
• https://github.com/torvalds/linux/tree/master/tools/perf source code in kernel

If the overhead is a problem, these tools can be improved. If a tool doesn't already, it could be rewritten in C to use perf_events_open() and mmap() for the trace buffer. It could also implement frequency counts in C, and operate on mmap() directly, rather than using awk/Perl/Python. Additional improvements are possible for ftrace-based tools, such as use of snapshots and per-instance buffers.

Some of these tools are intended as short-term workarounds until more kernel capabilities exist, at which point they can be substantially rewritten. Older versions of these tools will be kept in this repository, for older kernel versions.

Extra Profiling tools

• Java has HPROF http://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html

PMU

• https://gist.github.com/dannas/1fa2cfb0d3d108282955
• [ ] TODO: other links from the vbox issue #13
• https://www.usenix.org/legacy/event/hotcloud10/tech/full_papers/Du.pdf

[at15]

Fedora's perf does not include tips.txt result in warning when using perf report, the line should be at https://apps.fedoraproject.org/packages/perf/sources/

L1907 %doc linux-%{KVERREL}/tools/perf/Documentation/examples.txt

[at15]

perf has a flame graph https://github.com/brendangregg/FlameGraph

[at15]

golang has a tool that can be used with perf https://github.com/google/pprof

[at15]

linux jvm perf, flame graph http://batey.info/docker-jvm-flamegraphs.html

[at15]

redhat, also suggest a lot of performance monitoring stuff http://www.breakage.org/2014/06/06/getting-started-with-performance-analysis-of-docker/

[at15]

you can also run Spark on Mesos http://spark.apache.org/docs/latest/running-on-mesos.html

[at15]

a talk mentioned by java docker flame graph https://www.youtube.com/watch?v=7PkkxDaFDj8

• 21:44 unfold all
• 24:31 Default Java Flame Graphs with PMA is not helpful, red part is system and yellow part is Java
  • where is the stack
  • JVM JIT Compiler uses RBP
  • perf expects RBP to be the frame pointer
  • no frame pointer -> broken stacks
  • -XX:+PreserveFramePointer patch in OpenJDK, from 8u60
• 30:25 Where's the inlined frames
  • perf walks the real stack
  • PMA + unflodall + FlameGraphs + inlinedframes -> WIN!
• 36: 00 Skid
  • Super Scalar CPU
  • Speculative execution
  • Signal latency
  • The blamed instruction is often shortly after where the big cost lies
  • PC -> BCI -> LOC, you find the closest
• 45: 33 JMH perfasm
  • If you benchmark -> JMH FTW
  • JMH perf profilers:
    • perf: perf stat -d for your benchmark
    • perfnorm: normalized per op perf stat -d
    • perfasm: perf annotat for hot regions + top methods
  • http://psy-lob-saw.blogspot.com/2015/07/jmh-perfasm.html
• 50: 00 Summary/Suggested Work flow
  • Application level measurement
  • Application level analysis with FlameGraph
  • Identify and drill into bottlenecks
  • Capture in JMH benchmark (the real frame)
  • Iterate until improved
  • Repeat application level measurement/profile

[at15]

for monitoring a long running process, you can add sleep ... though I don't understand why

https://perf.wiki.kernel.org/index.php/Tutorial Attaching to a running process

ps ax | fgrep sshd

 2262 ?        Ss     0:00 /usr/sbin/sshd -D
 2787 pts/0    S+     0:00 fgrep --color=auto sshd

perf stat -e cycles -p 2262 sleep 2

 Performance counter stats for process id '2262':

    <not counted> cycles

       2.001263149  seconds time elapsed

TODO: but why the result it not counted ...

because the time is too short

sudo perf stat -p 17025 sleep 100 make the sleep longer and it will work, though I don't know if the sleep is the shell's sleep or something built into perf

[at15]

google has pprof

• https://github.com/google/pprof
• https://github.com/google/perf_data_converter