search

hashgraph / hedera-block-node

New Block Node services
Apache License 2.0
22 stars 2 forks source link

To report low-level Linux profiling of hedera-node under load tests #132

Open alex-kuzmin-hg opened 2 months ago

alex-kuzmin-hg commented 2 months ago

To setup, automate and regularly run Linux OS profiling reports of hedera node under various load tests

alex-kuzmin-hg commented 2 months ago

How to: 1) build own Perf locally, e.g. export PERF_EXEC_PATH=/home/alex.kuzmin/workspaces/linux/tools/perf 2) run test app sudo perf record -a -g -F 500 -- /usr/local/java/jdk-21.0.4+7/bin/java -XX:+UnlockDiagnosticVMOptions -XX:+PreserveFramePointer -XX:+DumpPerfMapAtExit 3) sudo: $PERF_EXEC_PATH/perf script report flamegraph

Result is: file:///Users/alex.kuzmin/Documents/workspaces/reports/flamegraph.html

alex-kuzmin-hg commented 2 months ago

Next step: to run 2 hours Network NFT test and publish for review

alex-kuzmin-hg commented 2 months ago

I/O profiling: sudo sysctl kernel.perf_event_paranoid=-1 kernel.kptr_restrict=0 sh /home/alex.kuzmin/workspaces/addIOprobes.sh ulimit -n 200000 ulimit -f 1024000 $PERF_EXEC_PATH/perf record -g --call-graph=fp -e 'probe_libc:*' --exclude-perf -F 500 -p $PID

alex-kuzmin-hg commented 2 months ago

Example:

<scheduler Pces 3657871 [005] 1315516.218289:                 probe_libc:__write: (7faf35690870)
      7faf35690870 __GI___libc_write+0x0 (/usr/lib/x86_64-linux-gnu/libc.so.6)
      7faf33debdaa writeBytes+0x1ca (/usr/local/java/jdk-21.0.4+7/lib/libjava.so)
      7faf33de4727 Java_java_io_FileOutputStream_writeBytes+0x17 (/usr/local/java/jdk-21.0.4+7/lib/libjava.so)
      7faf1c7c6d7f void java.io.FileOutputStream.writeBytes(byte[], int, int, boolean)+0xdf (/tmp/perf-3657702.map)
      7faf1d673314 void com.swirlds.common.io.extendable.ExtendableOutputStream.flush()+0x174 (/tmp/perf-3657702.map)
      7faf1d6b3bcc java.lang.Object com.swirlds.platform.wiring.PlatformWiring$$Lambda/0x00007fae984ba518.apply(java.lang.Object, java.lang.Object)+0x22c (/tmp/perf-3657702.map)
      7faf1cb50468 java.lang.Object com.swirlds.common.wiring.component.ComponentWiring$$Lambda/0x00007fae9859c440.apply(java.lang.Object)+0x68 (/tmp/perf-3657702.map)
      7faf1cac637c void com.swirlds.common.wiring.wires.input.BindableInputWire$$Lambda/0x00007fae984ae890.accept(java.lang.Object)+0xdc (/tmp/perf-3657702.map)
      7faf1c9f43fc void com.swirlds.common.wiring.schedulers.internal.SequentialThreadTaskScheduler.run()+0x8fc (/tmp/perf-3657702.map)
alex-kuzmin-hg commented 1 month ago

Example of file I/O profile (writing): some node (not a teacher/learner) for several minutes during NFT test: https://perf.analytics.eng.hashgraph.io/ephemeral/IO_profile.tmp/3/node_flamechart.svg https://perf.analytics.eng.hashgraph.io/ephemeral/IO_profile.tmp/3/node_flamechart_bytes.svg Heavy-loaded Learners/Teachers crash perf with such I/O profile. I'm working on it...

alex-kuzmin-hg commented 1 month ago

Reviewed with Jasper/Oleg