performance: please add profiling and send over profiles for analysis

[odeke-em]

Summary

To increase throughput, figure out what to optimize and figure out what code to focus, kindly please send me profiles

Problem Definition

We can't fix what we can't measure. Performance of a system is directly tied to security but also monetary gains for the system. Without profiling I can't be of adequate service to Quicksilver; some of the benefits of profiling are fixing pathologically slow code such as in the cosmos-sdk per PR https://github.com/cosmos/cosmos-sdk/pull/8719.

How to profile

Step 1: add net/http/pprof into cmd/quicksilverd

Firstly plus stop your running quicksilverd. Next we shall add profiling hooks into a patch of quicksilverd. To do so please edit the file cmd/quicksilverd/main.go and make it look like this having added our pprof import and implement this diff

diff --git a/cmd/quicksilverd/main.go b/cmd/quicksilverd/main.go
index 01283a4..60a015a 100644
--- a/cmd/quicksilverd/main.go
+++ b/cmd/quicksilverd/main.go
@@ -1,6 +1,8 @@
 package main

 import (
+   "net/http"
+   _ "net/http/pprof"
    "os"
    "path/filepath"

@@ -13,6 +15,12 @@ import (
 )

 func main() {
+   go func() {
+       if err := http.ListenAndServe(":6060", nil); err != nil {
+           panic(err)
+       }
+   }()
+
    setupConfig()
    cmdcfg.RegisterDenoms()

Please now run make install and get your node running again. After say 10 seconds once it is running great, please proceed onto the next step.

Step 2: Collect profiles

Please add this code below to a file called collect_profiles.go

package main

import (
    "archive/zip"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "path/filepath"
    "time"
)

func main() {
    dir := "quicksilverd_profiles"
    if err := os.MkdirAll(dir, 0755); err != nil {
        panic(err)
    }

    zipfileName := filepath.Join(dir, "profiles.zip")
    outzip, err := os.Create(zipfileName)
    if err != nil {
        panic(err)
    }
    defer func() {
        if err := outzip.Close(); err != nil {
            panic(err)
        }
        println("Please submit the zip file", zipfileName)
    }()

    zw := zip.NewWriter(outzip)
    if err != nil {
        panic(err)
    }
    defer zw.Close()

    const durInSeconds = 67 // 1min7seconds
    const url = "http://localhost:6060/debug/pprof"
    for i, n := 0, 60; i < n; i++ {
        // 1. Send a request for the CPU profile
        cpuProfile, err := fetchBody(url+"/profile", durInSeconds)
        if err != nil {
            panic(err)
        }
        ramHeapProfile, err := fetchBody(url+"/heap", durInSeconds)
        if err != nil {
            panic(err)
        }
        if err := writeOutToZips(zw, i+1, cpuProfile, ramHeapProfile); err != nil {
            panic(err)
        }

        fmt.Printf("Completed %d/%d profiles\r", i+1, n)
        time.Sleep(47 * time.Second)
    }
}

func writeOutToZips(zw *zip.Writer, fileIndex int, cpuProfile, ramProfile []byte) error {
    baseName := fmt.Sprintf("%d", fileIndex)
    wcf, err := zw.Create(baseName + ".cpu")
    if err != nil {
        return err
    }
    if _, err := wcf.Write(cpuProfile); err != nil {
        return err
    }

    wmf, err := zw.Create(baseName + ".mem")
    if err != nil {
        return err
    }
    if _, err := wmf.Write(ramProfile); err != nil {
        return err
    }
    return nil
}

func fetchBody(url string, durInSeconds int) ([]byte, error) {
    url += fmt.Sprintf("?seconds=%d", durInSeconds)
    res, err := http.Get(url)
    if err != nil {
        return nil, err
    }
    defer res.Body.Close()

    if code := res.StatusCode; code < 200 && code >= 300 {
        return nil, fmt.Errorf("%v failed to get a 2XX=>%d", res.Status, code)
    }

    return ioutil.ReadAll(res.Body)
}

and then run it by go run collect_profiles.go it should run for about ~38 minutes or so and each time it'll print out the progress. The profiles will be stored in a zip file under quicksilverd_profiles/profiles.zip

Step 3: Send over your .zip file with the 60 profiles

Send your profiles over to Orijtech Inc. Please upload the zip file quicksilverd_profiles/profiles.zip and email to emmanuel@orijtech.com. You can even increase the number of profiles to collect to collect data over a longer period

/cc @faddat @joe-bowman

---

For Admin Use

• [x] Not duplicate issue
• [ ] Appropriate labels applied
• [ ] Appropriate contributors tagged/assigned

[odeke-em]

@joe-bowman @tropicaldog @faddat how are we doing here? These profiles can massively help in directing us on what to fix in Quicksilver and what to make much faster.