Rolling Thread Dump / Profiling

[swirlds-automation]

Create a low overhead tool that produces rolling thread dumps. If cheap enough, this code should be enabled even on production environments.

[swirlds-automation]

2023-02-13 Discussion Notes between @edward-swirldslabs, @cody-littley, and @OlegMazurov

Development of an integrated continuous stack trace & thread dump utility to replace the manual use of jstack. We want the benefits of thread dumps without the cost of a stop-the-world snapshot.

Purpose

• Deadlock Detection
• Thread Profiling

Willing To Sacrifice

• Consistency/Completeness: since we're not stopping the world to collect a complete snapshot, some threads may start and some may stop and some may evade capture.
• Assumption: iterating over a list of open threads one at a time to collect stack traces is sufficient to both detect deadlocks and profile threads and code-paths to a sufficient granularity.
• Focus on application threads.

Requirements

• Implementation must be 100% stable and reliable, not interfering with the behavior or performance of the application
• Stack Trace collection must be run on its own thread.
• Dynamically Configurable.
• Just data collection, no analysis in the tool.
• Separate log files and configurable with time and size bounds for log rotation.
• Logs can easily be fed to appropriate analysis tools.

Dynamic Configuration / Tunable Parameters

• On / Off
• Granularity / Frequency of Collection Attempts
• Log rotation parameters: Time/Size

Future Possible Optimization

• Custom Binary Log Format
• Deduplication of Stack Trace Information in Log.
• Filters? (was not part of discussion)

Development Process

• Research / Discussion
• • [ ] Stack Trace Collection Methodologies
• • [ ] Log Format to enable data translation into analysis tools.
• Design
• • [ ] In-Memory Data Structures
• • [ ] Dynamic Configuration
• • [ ] Thread Iteration Algorithm
• Prototype Implementation
• • [ ] Thread Iteration, Stack Trace Collection, Logging
• • [ ] Stakeholder Review / FTV
• • [ ] Implementation Refinement to Analysis Ingestion
• MVP Implementation
• • [ ] Minimum Features Determined
• • [ ] Prototype Refined to MVP
• • [ ] Stakeholder Review / FTV

_{author:edward-swirldslabs, createdAt:2023-02-13T22:36:47Z, updatedAt=2023-02-13T22:36:47Z}

[swirlds-automation]

Slack Post from Oleg:

A couple of SO posts as examples of what we need to evaluate below. Once we are satisfied with what we get and the cost we can decide on output format/tooling for analysis. Here is an example output of jstack:

<adv crypto: merkle hash #2>" #239 daemon prio=5 os_prio=0 cpu=728474.61ms elapsed=4404.31s tid=0x00007f1e151c5730 nid=0x331ab0 runnable  [0x00007f210b8fd000]
   java.lang.Thread.State: RUNNABLE
        at com.swirlds.common.merkle.crypto.internal.MerkleCryptoEngine.digestSync(MerkleCryptoEngine.java:120)
        at com.swirlds.common.merkle.crypto.MerkleCryptography.digestSync(MerkleCryptography.java:59)
        at com.swirlds.common.merkle.hash.MerkleHashBuilder.hashSubtree(MerkleHashBuilder.java:220)
        - locked <0x00001008a7413e48> (a com.swirlds.merkle.tree.MerkleTreeInternalNode)
        at com.swirlds.common.merkle.hash.MerkleHashBuilder.lambda$createHashingRunnable$1(MerkleHashBuilder.java:179)
        at com.swirlds.common.merkle.hash.MerkleHashBuilder$$Lambda$789/0x0000000801077510.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.5/ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.5/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base@17.0.5/Thread.java:833) ```

I care about thread name, cpu=, elapsed=, Thread.State, stack trace, locked/waiting info with object ids. Can we get those programmatically with low overhead? Links: https://stackoverflow.com/questions/36700586/how-to-capture-thread-dump-programatically-using-java-code https://stackoverflow.com/questions/12965836/trigger-complete-stack-dump-programmatically _{author:edward-swirldslabs, createdAt:2023-02-13T22:38:41Z, updatedAt=2023-02-13T22:38:41Z}

[swirlds-automation]

_{migrated from: url=https://github.com/swirlds/swirlds-platform/issues/6031 author:cody-littley, #:6031, createdAt:2022-09-15T20:50:16Z, updatedAt=2023-02-17T22:23:26Z labels=New Feature,Backlog,On Ramp,Migration:Hashgraph}