Proposal (Gnovm / Gno test): Implement `gno test -profile`

[thinhnx-var]

Introduction

This proposal outlines the implementation of a new flag, -profile, for the gno test command. This flag will enable developers to profile the execution of their code written in Gno, providing valuable insights into performance characteristics.

Motivation

Optimizing smart contract performance is crucial for reducing gas costs and improving user experience. Currently, there's a lack of dedicated profiling tools within the Gno ecosystem. This proposal aims to introduce a built-in profiling functionality for gno test.

Proposed Approach

The gno test -profile flag will initiate profiling during gno file test execution. Here are two potential implementation strategies:

1. On-Chain Instrumentation:

During the Gno compilation process, inject profiling code into the smart contract bytecode. This code will track metrics like function execution times and gas consumption during test execution on the blockchain. After test completion, the profiling data will be retrieved and aggregated for analysis.

Due to its broader scope and potential impact on various aspects of the system, this implementation falls outside the boundaries of this proposal.

2. Off-Chain Instrumentation:

Leverage existing testing frameworks to capture performance data during test execution within the Gno environment. This data could include:

• function execution times
• gas consumption
• allocated memmory
• VM cycles used
• etc.

that will be measured locally. The profiling data will be stored and presented within the test results.

Technical Considerations:

Performance Overhead: The profiling process should have minimal impact on the actual execution time of the smart contract being tested. Data Granularity: The level of detail exposed in the profiling data needs to balance usefulness with potential privacy concerns.

Success Criteria:

• The gno test -profile flag successfully captures and presents profiling data during gno testing.
• The profiling data provides valuable insights into Gno code performance characteristics.
• The implementation has minimal impact on the overall performance of the gno test command.

Conclusion

The introduction of gno test -profile will empower Gno developers with a valuable tool for gno code performance analysis. This will contribute to the development of more efficient and cost-effective smart contract applications on the Gno platform.

Question

Which indexes, informations should (and should not) be exposed during gno test -profile ?

We welcome comments and suggestions from the Gno community to make this proposal even stronger.

[linhpn99]

i think it's good for long term

[thinhnx-var]

Maybe we can have this format on a .test_result file or in the standard test output.

[thehowl]

Let's not reinvent the wheel and re-use pprof as much as possible. There is some documentation on how to use the proto file to make profiling.

Creating profiling formats and systems is hard. Creating the tooling to make sense of the profiling results is harder. Go developers are used to pprof, so it makes sense to reuse that, also because it already has a plethora of good visualization options.

In the meantime, let's make this work only off-chain, as a debugging tool to see where the most VM cycles are being spent, most time allocated, etc. We can then have options to enable it on a node, although I see this as less urgent for the time being.

In the implementation, try not to impact run-time performance too much and adding code on hot paths. There are some benchmarks in pkg/gnolang you can run to see performance in the gnovm, very broadly speaking. You can run them as follows:

CGO_ENABLED=0 go test -bench Benchdata -benchmem -short -run=^$ -cpu 1 -count=1 ./gnovm/pkg/gnolang/

Please compare the results with master using benchstat.

Please touch base regularly with what you're doing and get feedback early on any design decisions. Ping us on signal if we don't answer on GitHub. Make small PRs as much as possible, use stacking. (In spite of what the page says, you just need to make PRs that are based off the work of the previous ones, so instead of doing git checkout -b on master you'll do it on what you're already working on.)

UX-wise, re-use what Go has as much as possible. Ie. -cpuprofile and -memprofile for gno test.

cc @mvertes for an additional pair of eyes. You can proceed with drafting a technical outline of how this would work (ie. what needs to be added to the gnovm, what changes to do on the CLIs, etc. -- make a plan, this top-level comment does not clarify what you want to do yet.)

[mvertes]

Good profiling is so difficult to implement and so easy to get wrong, so yes, Morgan is right, it is absolutely necessary to consider at least the existing profiling infrastructure already provided by the Go tool chain, which is a top one, even despite issues as in go1.18 golang/go#52912, for example.

Possible alternatives:

• to bake your own sampler in GnoVM or event capture or analysing and visualising tools, you will have something incomplete, providing partial infos (no low-level, no system), probably slow (defeating the purpose, adding more problems than solutions), and anyway involving a lot of replicated efforts.
• to build something based on eBPF, or dtrace, which provides sampling at system level, and user level probes inserted in apps. In addition of not being portable and hard to deploy, the information you will get will be partial, as the Go runtime (in addition of Gno) will be opaque: you will see threads but no goroutines, heap alloc/dealloc but no garbage-collect information, etc.

So let's consider now only the extension of existing Go pprof.

To have, let's say, a CPU profiler at GnoVM level, here is what I would consider:

• Extend the pprof sampling to capture stack traces at GnoVM level, in addition to low level stack traces: without this information it is simply impossible to relate which Gno source code is involved. This information has to be acquired during execution and can not be reconstructed offline. Stack traces are also useful for memory profiling, so it's not a wasted effort. Having both the GnoVM level and the binary level stacks in the same sample will also permit to correlate more data and uncover more optimization opportunities in the VM implementation. For information, this is exactly how the embedded profiler of Google javascript V8 is architected: sampling stack traces both at C/C++ level and JavaScript VM level.
• Extend the pprof symbolizer to resolve the GnoVM stack traces (stack traces contain only low-level basic info: integers for date, counters, etc. and memory pointers for functions and frames. This is in the part of pprof which is offline so performance is not critical here.

This has to be done while preserving the compatibility of data structures with the profiling analysis and visualisation systems (See pprof protocol definitions).

Most of it also apply in big parts to memory profiling as well.

Given the current maturity of the GnoVM, it's probably a bit early stage to consider this now. I would advise to wait for the GnoVM to be more stabilised. For example that capturing stack traces would cost a lot, due to the current architecture, as opposed for example to a regular byte code based virtual machine, which has a flatter memory representation.

But anyway, this is certainly an interesting topic, and thank you for launching this discussion!

[thinhnx-var]

I've pored over the comments a bunch of times, and tried to get all of it, so I will wait till I can control mostly things (like the eBPF and dtrace), have a deeper view on the GnoVM's architecture to propose something new.

Thank you for sharing your though on this topic. It is really help me and others so much.