bingoohuang commented 5 years ago

Golang多版本安装

method1 官方

$ go get -u golang.org/dl/go1.12.3
go: finding golang.org/dl latest
go: downloading golang.org/dl v0.0.0-20190408222801-b337094d5ff3
go: extracting golang.org/dl v0.0.0-20190408222801-b337094d5ff3
$ go1.12.3 download
Downloaded 100.0% (127615731 / 127615731 bytes)
Unpacking /Users/bingoobjca/sdk/go1.12.3/go1.12.3.darwin-amd64.tar.gz ...
Success. You may now run 'go1.12.3'
$ go1.12.3 version
go version go1.12.3 darwin/amd64
$ go version
go version go1.12.1 darwin/amd64

无缓冲读

try online

package main

import "fmt"

func main() {
    // create unbuffered channel of int values with capacity of 1
    ch := make(chan int)

    select {
        case ch <- 3: 
        // uncomment the following line to get this program work
        // default:
    }

    fmt.Printf("ok\n")
}

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan send]:
main.main()
    /tmp/056148531/main.go:10 +0x60
exit status 2

bingoohuang commented 5 years ago

一些库

Go 语言 Excel 类库 Excelize
Kratos是bilibili开源的一套Go微服务框架，包含大量微服务相关框架及工具。

bingoohuang commented 5 years ago

一些工具链

dogsleda Go static analysis tool to find assignments/declarations with too many blank identifiers (e.g. x, , , _, := f()). Its name was inspired from this reddit post.

bingoohuang commented 5 years ago

包管理，经常看到go.mod文件中，有一些indirect的包，是怎么被依赖进来的，可以用命令go mod why yourAwesomePackage来查看。比如：

# bingoo @ 192 in ~/GitHub/loglineparser on git:master x [12:40:02]
$ go mod why gopkg.in/yaml.v2
# gopkg.in/yaml.v2
github.com/bingoohuang/loglineparser
github.com/araddon/dateparse
github.com/araddon/dateparse.test
github.com/simplereach/timeutils
gopkg.in/mgo.v2/bson
gopkg.in/mgo.v2/bson.test
gopkg.in/yaml.v2

StackOverflow上的一篇帖子

As of the final 1.11 release, the go module cache (used for storing downloaded modules and source code), is in the $GOPATH/pkg/mod location (see the docs here). For clarification, the go build cache (used for storing recent compilation results) is in a different location.

This article, indicated that it's in the $GOPATH/src/mod, but in the timespan of the recent ~40 days, the golang team must have changed that target location. This message thread has some discussion on why the downloaded items ended up in $GOPATH/pkg.

You can also use the go mod download -json command to see the downloaded modules/source metadata and their location on your local disk. Example output below:

$ go mod download -json go: finding github.com/aws/aws-sdk-go v1.14.5 go: finding github.com/aws/aws-lambda-go v1.2.0

{
    "Path": "github.com/aws/aws-lambda-go",
    "Version": "v1.2.0",
    "Info": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.info",
    "GoMod": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.mod",
    "Zip": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.zip",
    "Dir": "/go/pkg/mod/github.com/aws/aws-lambda-go@v1.2.0",
    "Sum": "h1:2f0pbAKMNNhvOkjI9BCrwoeIiduSTlYpD0iKEN1neuQ=",
    "GoModSum": "h1:zUsUQhAUjYzR8AuduJPCfhBuKWUaDbQiPOG+ouzmE1A="
}
{
    "Path": "github.com/aws/aws-sdk-go",
    "Version": "v1.14.5",
    "Info": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.info",
    "GoMod": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.mod",
    "Zip": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.zip",
    "Dir": "/go/pkg/mod/github.com/aws/aws-sdk-go@v1.14.5",
    "Sum": "h1:+l1m6QH6LypE2kL0p/G0Oh7ceCv+IVQ1h5UEBt2xjjU=",
    "GoModSum": "h1:ZRmQr0FajVIyZ4ZzBYKG5P3ZqPz9IHG41ZoMu1ADI3k="
}

bingoohuang commented 5 years ago

JSON

JSON数据解析，直接给字段赋予json字符串，避免转义

run

package main

import (
    "encoding/json"
    "fmt"
)

func main() {
    s := `[` + `{"name":"bingoo"},{"name":"dingoo"}` + `]`
    var arr []interface{}
    if err := json.Unmarshal([]byte(s), &arr); err != nil {
        panic(err)
    }

    m := map[string]interface{}{"key1": arr, "key2": s, "key3": json.RawMessage([]byte(s))}
    jso, err := json.Marshal(m)
    if err != nil {
        panic(err)
    }

    // {"key1":[{"name":"bingoo"},{"name":"dingoo"}],"key2":"[{\"name\":\"bingoo\"},{\"name\":\"dingoo\"}]","key3":[{"name":"bingoo"},{"name":"dingoo"}]}
    fmt.Println(string(jso))
}

JSON解析库

流式JSON解析库 GoJay aims to be a very fast, JIT stream parser with 0 reflection, low allocation with a friendly API.

Alternate implementations

Popular replacements for standard library packages:

encoding/json -> ffjson, easyjson, jingo (only encoder), etc
net/http
- fasthttp (but incompatible API, not RFC compliant in subtle ways)
- httprouter (has other features besides speed; I've never actually seen routing in my profiles)
regexp -> ragel (or other regular expression package)
serialization
- encoding/gob -> alecthomas/go_serialization_benchmarks
- protobuf -> gogo/protobuf
- all serialization formats have trade-offs: choose one that matches what you need
  - Write heavy workload -> fast encoding speed
  - Read-heavy workload -> fast decoding speed
  - Other considerations: encoded size, language/tooling compatibility
- tradeoffs of packed binary formats vs. self-describing text formats
database/sql -> has tradeoffs that affect performance
- look for drivers that don't use it: jackx/pgx, crawshaw sqlite, ...
gccgo (benchmark!), gollvm (WIP)
container/list: use a slice instead (almost always)

gojsonq A simple Go package to Query over JSON Data. It provides simple, elegant and fast ODM like API to access, query JSON document

import "github.com/thedevsaddam/gojsonq"

func main() {
    const json = `{"name":{"first":"Tom","last":"Hanks"},"age":61}`
    name := gojsonq.New().FromString(json).Find("name.first")
    println(name.(string)) // Tom
}

bingoohuang commented 5 years ago

强制确保类型实现某个接口

Go语言中，类型实现某个接口，只要实现了该接口中所有定义的方法即可，没有像Java中的implements关键字，是一种契约式精神的体现，或者说鸭子类型。那有没有强制某个类型必须某个接口的写法呢，刚刚在翻阅jons.RawMessage中看到了以下两行代码，刚开始感觉迷惑，仔细思考后，便了解了其意图：

var _ Marshaler = (*RawMessage)(nil)
var _ Unmarshaler = (*RawMessage)(nil)

带上下文的代码如下：

// RawMessage is a raw encoded JSON value.
// It implements Marshaler and Unmarshaler and can
// be used to delay JSON decoding or precompute a JSON encoding.
type RawMessage []byte

// MarshalJSON returns m as the JSON encoding of m.
func (m RawMessage) MarshalJSON() ([]byte, error) {
    if m == nil {
        return []byte("null"), nil
    }
    return m, nil
}

// UnmarshalJSON sets *m to a copy of data.
func (m *RawMessage) UnmarshalJSON(data []byte) error {
    if m == nil {
        return errors.New("json.RawMessage: UnmarshalJSON on nil pointer")
    }
    *m = append((*m)[0:0], data...)
    return nil
}

var _ Marshaler = (*RawMessage)(nil)
var _ Unmarshaler = (*RawMessage)(nil)

bingoohuang commented 5 years ago

在for循环里不要使用select + time.After的组合，有坑

本来想给朋友宣称一下，西游可能有Bug，我都已经被挤下线了，但是还能继续谷歌（假百度），然后随便输入了一个golang oom作为搜索条件：

结果就把这个搜索结果页面扔到一边，去干其他事情去了。

等回过头来，打算关闭这个标签页的时候，发现结果第一条，还是有点兴趣的，就点进去看了，最后有这么一句话：

有经验的gopher都知道，在for循环里不要使用select + time.After的组合，有坑。当使用golang过程中，遇到性能和内存gc问题，都可以使用golang tool pprof来排查分析问题。

然后，想到自己也经常这么用（for select time.After），很可能踩坑。然后去继续探索了一下，为什么可能会有坑，然后将代码中的for select time.After全部重构成 time.Timer + for来预防坑。

Go 编程: 对不起，你的 CPU 泄露了

Don't use time.Tick to prevent leaks

就是因为在循环中，不停的创建新的计时器，而每个计时器都会开启内部协程。再看看计时器函数的官方注释：


// Tick is a convenience wrapper for NewTicker providing access to the ticking
// channel only. While Tick is useful for clients that have no need to shut down
// the Ticker, be aware that without a way to shut it down the underlying
// Ticker cannot be recovered by the garbage collector; it "leaks".
// Unlike NewTicker, Tick will return nil if d <= 0.
func Tick(d Duration) <-chan Time {
    if d <= 0 {
        return nil
    }
    return NewTicker(d).C
}

bingoohuang commented 5 years ago

一些代码检查工具

golangci-lint run, 来自这里，安装GO111MODULE=on go get github.com/golangci/golangci-lint/cmd/golangci-lint@v1.16.0
一些定制 golangci-lint run --exclude="cyclomatic complexity" --exclude-use-default=false --enable=golint --enable=gocyclo --enable=goconst --enable=unconvert ./...
golangci-lint run --enable-all
Staticcheck – a collection of static analysis tools for working with Go code，安装GO111MODULE=off go get honnef.co/go/tools/cmd/...，运行staticcheck ./...
像牛人一样改进你的Go代码
Go Report Card,golang进阶:怎么开发一个热门的开源项目
A tool to list and diagnose Go processes currently running on your system

bingoohuang commented 5 years ago

golang进阶:go1.12 mod 教程

bingoohuang commented 5 years ago

检查依赖是否有升级

go get -u github.com/psampaz/go-mod-outdated
go list -u -m -json all | go-mod-outdated -direct

example output

+---------------------------------+----------------------------------------------------------------+------------------------------------+--------+------------------+
|             MODULE              |                            VERSION                             |            NEW VERSION             | DIRECT | VALID TIMESTAMPS |
+---------------------------------+----------------------------------------------------------------+------------------------------------+--------+------------------+
| github.com/alecthomas/template  | v0.0.0-20160405071501-a0175ee3bccc                             |                                    | true   | true             |
| github.com/bingoohuang/gou      | v0.0.0-20190604082926-bf3d9b2b55aa5840c442284656ed6b15aedc5a25 | v0.0.0-20190604082926-be6100942b5a | true   | false            |
| github.com/bingoohuang/now      | v0.0.0-20190604021600-70970d3ad0e7                             |                                    | true   | true             |
| github.com/bingoohuang/statiq   | v0.2.1                                                         |                                    | true   | true             |
| github.com/dgraph-io/badger     | v2.0.0-rc.2+incompatible                                       | v1.5.4                             | true   | false            |
| github.com/gchaincl/dotsql      | v0.1.0                                                         |                                    | true   | true             |
| github.com/gin-contrib/sessions | v0.0.0-20190226023029-1532893d996f                             | v0.0.0-20190512062852-3cb4c4f2d615 | true   | true             |
| github.com/gin-gonic/gin        | v1.4.0                                                         |                                    | true   | true             |
| github.com/go-sql-driver/mysql  | v1.4.1                                                         |                                    | true   | true             |
| github.com/lib/pq               | v1.0.0                                                         | v1.1.1                             | true   | true             |
| github.com/mattn/go-sqlite3     | v1.10.0                                                        |                                    | true   | true             |
| github.com/patrickmn/go-cache   | v2.1.0+incompatible                                            |                                    | true   | true             |
| github.com/pkg/errors           | v0.8.1                                                         |                                    | true   | true             |
| github.com/sirupsen/logrus      | v1.4.2                                                         |                                    | true   | true             |
| github.com/spf13/pflag          | v1.0.3                                                         |                                    | true   | true             |
| github.com/spf13/viper          | v1.3.2                                                         | v1.4.0                             | true   | true             |
| github.com/stretchr/testify     | v1.3.0                                                         |                                    | true   | true             |
| github.com/swaggo/swag          | v1.4.1                                                         | v1.5.0                             | true   | true             |
| github.com/thoas/go-funk        | v0.4.0                                                         |                                    | true   | true             |
+---------------------------------+----------------------------------------------------------------+------------------------------------+--------+------------------+

bingoohuang commented 5 years ago

Golang中的对象健美操

One level of indentation per method.
Don't use the ELSE keyword.
Wrap all primitives and Strings in classes.
First class collections.
One dot per line.
Don't abbreviate.
Keep all classes less than 50 lines.
No classes with more than two instance variables.
No getters or setters.

bingoohuang commented 5 years ago

Go linux安装

下载，解压缩

[vagrant@bogon ~]$ curl -O https://dl.google.com/go/go1.12.5.linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  122M  100  122M    0     0   349k      0  0:05:57  0:05:57 --:--:--  356k
[vagrant@bogon ~]$ tar -xzf go1.12.5.linux-amd64.tar.gz
[vagrant@bogon ~]$ ls
base.sh  cleanup.sh  go  go1.12.5.linux-amd64.tar.gz  puppet.sh  vagrant.sh  virtualbox.sh  zerodisk.sh

设置环境变量PATH

[vagrant@bogon ~]$ echo "export PATH=\$PATH:/home/vagrant/go/bin" >> ~/.bashrc; source ~/.bashrc
[vagrant@bogon ~]$ go version
go version go1.12.5 linux/amd64

bingoohuang commented 5 years ago

Go1.13将正式开始Go2开发历程

首先是执行go get golang.org/dl/gotip安装tip的辅助命令，然后通过执行gotip download下载真正的tip版本工具。下载完成之后，就可以通过totip命令来编译和运行Go程序了。

➜  kerb git:(master) ✗ go get golang.org/dl/gotip
go: finding golang.org/dl/gotip latest
go: finding golang.org/dl latest
go: downloading golang.org/dl v0.0.0-20190507014322-219d744c5398
go: extracting golang.org/dl v0.0.0-20190507014322-219d744c5398
➜  kerb git:(master) ✗ gotip download
Cloning into '/Users/bingoobjca/sdk/gotip'...
remote: Counting objects: 9475, done
remote: Finding sources: 100% (9475/9475)
remote: Total 9475 (delta 1010), reused 5862 (delta 1010)
package main
Receiving objects: 100% (9475/9475), 22.11 MiB | 363.00 KiB/s, done.
Resolving deltas: 100% (1010/1010), done.
Checking out files: 100% (8598/8598), done.
HEAD is now at f2a4c13 errors: clarify doc for As
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
---
Installed Go for darwin/amd64 in /Users/bingoobjca/sdk/gotip
Installed commands in /Users/bingoobjca/sdk/gotip/bin
Success. You may now run 'gotip'!
➜  kerb git:(master) ✗ gotip version
go version devel +f2a4c13 Tue Jun 11 21:50:05 2019 +0000 darwin/amd64

bingoohuang commented 5 years ago

GOPROXY

# Enable the go modules feature
export GO111MODULE=on
# Set the GOPROXY environment variable
export GOPROXY=https://goproxy.io

proxy.golang.org

export GO111MODULE=on
export GOPROXY=https://proxy.golang.org

from go 1.13

GOPROXY=direct,https://127.0.0.1:12333,https://goproxy.cn,https://goproxy.io,https://mirrors.aliyun.com/goproxy,https://athens.azurefd.net
go env -w GOSUMDB="off"

bingoohuang commented 5 years ago

Go 语言诞生时，我们称它为系统编程语言，我有点遗憾，因为很多人因此认为它是一种操作系统编写语言。我们应该称它为服务编写语言，这是我们真正想做的。现在我想明白了，Go 是云基础架构语言，因为系统编程的另一个定义是云中运行的东西。

When we first announced Go, we called it a systems programming language, and I slightly regret that because a lot of people assumed it was an operating systems writing language. What we should have called it is a server writing language, which is what we really thought of it as. Now I understand that what we have is a cloud infrastructure language. Another definition of systems programming is the stuff that runs in the cloud.

-- Rob Pike

bingoohuang commented 5 years ago

今天我探索了一下golang的两个方面：

类似于Java中有很好的Builder模式
Getter和Setter的命名约定

然后我发现了函数选项（Functional Options）模式， GIST上有一个最小的例子。

函数选项模式是由Rob Pike提出，并由Dave Cheney等推广开，它优雅地解决了go语言中默认参数问题。

下面总结一下，函数选项模式有哪些优点：

支持默认参数：不必向结构体参数那样，不使用时仍必须传递一个空的struct值
代码简洁：即使是像go-micro这种支持如此繁多选项，代码也很美观
扩展性好：增加新的选项只需少量代码

推而广之：类似结构体中变量的赋值都可以效仿之。

Using functional options instead of method chaining in Go中，以gorm为示例，使用函数选项为例，改造了gorm的用法。
Fluent Middleware in golang 使用了类似的Fluent模式。
类型安全的Reusable and type-safe options for Go API

bingoohuang commented 5 years ago

溜开源项目

bingoohuang commented 5 years ago

创建的对象，里面已经启动了一个后台的死循环的go协程，当对象不再被使用时，因为背后的go协程还一直在跑，导致对象不能被gc回收，咋办？，参见go-cache的实现

func New(defaultExpiration, cleanupInterval time.Duration) *Cache {
    items := make(map[string]Item)
    return newCacheWithJanitor(defaultExpiration, cleanupInterval, items)
}

func newCacheWithJanitor(de time.Duration, ci time.Duration, m map[string]Item) *Cache {
    c := newCache(de, m)
    // This trick ensures that the janitor goroutine (which--granted it
    // was enabled--is running DeleteExpired on c forever) does not keep
    // the returned C object from being garbage collected. When it is
    // garbage collected, the finalizer stops the janitor goroutine, after
    // which c can be collected.
    C := &Cache{c}
    if ci > 0 {
        runJanitor(c, ci)
        runtime.SetFinalizer(C, stopJanitor)
    }
    return C
}

func runJanitor(c *cache, ci time.Duration) {
    j := &janitor{
        Interval: ci,
        stop:     make(chan bool),
    }
    c.janitor = j
    go j.Run(c)
}

bingoohuang commented 4 years ago

以库为驱动开发的Golang工程结构，摘自Library driven development

Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library. Sometimes you might want users to interact in multiple ways so you create multiple binaries. For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version. You can easily do this by organizing your project like this:

adder/
  adder.go
  cmd/
    adder/
      main.go
    adder-server/
      main.go

Users can install your “adder” application binaries with “go get” using an ellipsis:

$ go get github.com/benbjohnson/adder/...

And voila, your user has “adder” and “adder-server” installed!

稍微复杂一点的结构，摘自Go project structure to produce library and cli with the same name in single repository

Of course if your project is more complex, you may create further packages under the project root, and it may have multiple commands (multiple main packages), e.g.:

host.com/project/
    cmd/
        project/
            project.go
        prjtool/
            prjtool.go
    packagex/
        x.go
    packagey/
        y.go
    filea.go
    fileb.go

An example following this layout is the very popular Go Delve debugger (4.5k stars currently).

bingoohuang commented 4 years ago

Passing callbacks and pointers to Cgo

This post discusses an end-to-end example that covers:

Basic usage of Cgo, including linking a custom C library into the Go binary.
Passing structs from Go to C.
Passing Go functions to C and arranging C to call them back later.
Safely passing arbitrary Go data to C code, which can later pass it back to the Go callbacks it invokes.

The full source code for this example is available on Github. cgo-callback.zip

bingoohuang commented 4 years ago

宽进严出原则：

Be conservative in what you send, be liberal in what you accept

Robustness Principle

在golang上的应用：

Return concrete types, receive interfaces as parameters

选哪个？

func New() *os.File
func New() io.ReadWriteCloser
func New() io.Writer
func New() interface{}

根据宽进严出: 选 func New() *os.File

摘自理解go interface

bingoohuang commented 4 years ago

看goroutine的数量

➜  golang-trial git:(master) ✗ GODEBUG=schedtrace=1000 gohttpd
SCHED 0ms: gomaxprocs=12 idleprocs=10 threads=4 spinningthreads=1 idlethreads=0 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 1009ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 2013ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 3019ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 4026ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 5033ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 6036ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=21 runqueue=152 [4 1 11 0 2 1 3 10 4 5 1 9]
SCHED 7042ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=209 [12 7 15 2 18 15 0 13 19 3 2 16]
SCHED 8046ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=170 [0 12 10 8 4 5 0 3 4 5 1 5]
SCHED 9055ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=21 runqueue=234 [3 0 15 9 3 11 1 10 5 6 10 3]
SCHED 10056ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=221 [1 13 15 9 5 2 0 0 8 12 4 10]
SCHED 11058ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=18 runqueue=159 [9 12 4 13 4 9 0 10 11 1 1 12]
SCHED 12061ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=204 [0 16 3 18 7 16 3 2 17 12 18 13]
SCHED 13062ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=1 idlethreads=23 runqueue=237 [16 12 3 13 7 6 0 1 1 15 9 21]
SCHED 14062ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=17 runqueue=201 [16 1 1 11 14 2 5 10 2 0 0 16]
SCHED 15072ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=22 runqueue=26 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 16080ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=21 runqueue=202 [10 16 18 0 12 0 19 13 9 17 0 1]
SCHED 17087ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=15 runqueue=167 [8 3 12 14 14 14 4 17 1 6 15 6]
SCHED 18092ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=209 [18 4 0 17 4 9 0 10 0 3 5 16]
SCHED 19093ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=21 runqueue=198 [16 17 3 6 4 1 14 6 18 18 10 21]
SCHED 20093ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=24 runqueue=232 [7 12 13 5 3 7 5 2 8 1 9 8]
SCHED 21100ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=24 runqueue=206 [16 1 14 9 16 7 16 13 5 1 8 4]
SCHED 22110ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=23 runqueue=249 [8 6 8 3 10 9 12 8 6 11 0 3]
SCHED 23113ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=20 runqueue=223 [0 5 17 2 5 3 0 0 0 14 9 2]
SCHED 24116ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=1 idlethreads=22 runqueue=219 [1 0 9 3 1 2 13 10 1 1 18 12]
SCHED 25124ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=23 runqueue=199 [14 12 15 6 2 1 5 0 11 1 15 0]
SCHED 26126ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=21 runqueue=169 [1 12 6 11 8 5 8 5 5 10 6 4]
SCHED 27126ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=20 runqueue=232 [4 5 9 0 1 1 1 1 5 1 11 4]
SCHED 28129ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=21 runqueue=243 [9 11 7 3 7 14 3 12 2 12 12 1]
SCHED 29136ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=20 runqueue=58 [0 4 0 1 0 0 2 0 0 3 0 0]
SCHED 30144ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=153 [0 9 7 1 4 2 10 2 5 0 0 1]
SCHED 31151ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=216 [10 4 9 5 14 7 1 10 7 18 8 5]
SCHED 32159ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=24 runqueue=203 [5 8 2 0 5 1 1 14 3 0 14 12]
SCHED 33164ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=212 [14 13 5 19 0 13 1 8 15 9 11 2]
SCHED 34174ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=23 runqueue=266 [6 4 12 12 1 0 8 4 0 3 0 3]

sched：每一行都代表调度器的调试信息，后面提示的毫秒数表示启动到现在的运行时间，输出的时间间隔受 schedtrace 的值影响。
gomaxprocs：当前的 CPU 核心数（GOMAXPROCS 的当前值）。
idleprocs：空闲的处理器数量，后面的数字表示当前的空闲数量。
threads：OS 线程数量，后面的数字表示当前正在运行的线程数量。
spinningthreads：自旋状态的 OS 线程数量。
idlethreads：空闲的线程数量。
runqueue：全局队列中中的 Goroutine 数量，而后面的 [0 0 1 1] 则分别代表这 4 个 P 的本地队列正在运行的 Goroutine 数量。

参考:

用 GODEBUG 看调度跟踪

bingoohuang commented 4 years ago

测试相关

TestMain

在写测试时，有时需要在测试之前或之后进行额外的设置（setup）或拆卸（teardown）；有时，测试还需要控制在主线程上运行的代码。为了支持这些需求，testing 提供了 TestMain 函数:

func TestMain(m *testing.M)

package mytestmain

import (  
    "flag"
    "fmt"
    "os"
    "testing"
)

var db struct {  
    Dns string
}

func TestMain(m *testing.M) {
    db.Dns = os.Getenv("DATABASE_DNS")
    if db.Dns == "" {
        db.Dns = "root:123456@tcp(localhost:3306)/?charset=utf8&parseTime=True&loc=Local"
    }

    flag.Parse()
    exitCode := m.Run()

    db.Dns = ""

    // 退出
    os.Exit(exitCode)
}

func TestDatabase(t *testing.T) {
    fmt.Println(db.Dns)
}

参数化测试

func TestAdd(t *testing.T) {
    tests := []struct{
        name     string
        first    int64
        second   int64
        expected int64
    } {
        {
            name:     "HappyPath":
            first:    2,
            second:   3,
            expected: 5,
        },
        {
            name:     "NegativeNumber":
            first:    -1,
            second:   -1,
            expected: -2,
        },
    }

    for _, test := range tests {
        t.Run(test.name, func(t *testing.T) {
            assert.Equal(t, test.expected, Add(test.first, test.second))
        })
    }
}

测试集

import (
    "testing"
    "github.com/stretchr/testify/suite"
)

type ExampleTestSuite struct {
    suite.Suite
    VariableThatShouldStartAtFive int
}

func (suite *ExampleTestSuite) SetupTest() {
    suite.VariableThatShouldStartAtFive = 5
}

func (suite *ExampleTestSuite) TestExample() {
    suite.Equal(suite.VariableThatShouldStartAtFive, 5)
}

func TestExampleTestSuite(t *testing.T) {
    suite.Run(t, new(ExampleTestSuite))
}

BDD

var _ = Describe("Book", func() {
    var (
        book Book
        err error
    )

    BeforeEach(func() {
        book, err = NewBookFromJSON(`{
            "title":"Les Miserables",
            "author":"Victor Hugo",
            "pages":1488
        }`)
    })

    Describe("loading from JSON", func() {
        Context("when the JSON fails to parse", func() {
            BeforeEach(func() {
                book, err = NewBookFromJSON(`{
                    "title":"Les Miserables",
                    "author":"Victor Hugo",
                    "pages":1488oops
                }`)
            })

            It("should return the zero-value for the book", func() {
                Expect(book).To(BeZero())
            })

            It("should error", func() {
                Expect(err).To(HaveOccurred())
            })
        })
    })
})

Mock

使用 gomock 提供的 mockgen 工具命令
使用 sqlmock 来模拟数据库的连接
httpmock 就是一个用于 Mock 所有 HTTP 依赖的包，它使用模式匹配的方式匹配 HTTP 请求的 URL，在匹配到特定的请求时就会返回预先设置好的响应。
猴子补丁其实就是一个大杀器了，bouk/monkey 能够通过替换函数指针的方式修改任意函数的实现，所以如果上述的几种方法都不能满足我们的需求，我们就只能够通过猴子补丁这种比较 hack 的方法 Mock 依赖了,：
```
func main() {
    monkey.Patch(fmt.Println, func(a ...interface{}) (n int, err error) {
        s := make([]interface{}, len(a))
        for i, v := range a {
            s[i] = strings.Replace(fmt.Sprint(v), "hell", "*bleep*", -1)
        }
        return fmt.Fprintln(os.Stdout, s...)
    })
    fmt.Println("what the hell?") // what the *bleep*?
}
```
不要在单元测试之外的地方使用猴子补丁，我们应该只在必要的时候使用这种方法，例如依赖的第三方库没有提供 interface 或者修改 time.Now 以及 rand.Int63n 等内置函数的返回值用于测试时。

从理论上来说，通过猴子补丁这种方式我们能够在运行时 Mock Go 语言中的一切函数，这也为我们提供了单元测试 Mock 依赖的最终解决方案。

bingoohuang commented 4 years ago

go module upgrades dependencies:

go get -u (without any arguments) now only upgrades the direct and indirect dependencies of your current package, and no longer examines your entire module.
go get -u ./... from your module root upgrades all the direct and indirect dependencies of your module, and now excludes test dependencies.
go get -u -t ./... is similar, but also upgrades test dependencies.

bingoohuang commented 4 years ago

Go Profiling and Optimization.pptx 一文读懂 Go profiling 和性能优化

火焰图进行性能分析.pptx

bingoohuang commented 4 years ago

GO语言的历史，非常好的演讲，值得一看。

编程语言发展的四波浪潮:

第一波浪潮：语言扩张 - 巴别塔特征：多样化。很久以前，语言是多种多样的，并在在思想、方法和意见等方面体现出多样性。 2 第二波浪潮：语言的标准化特征：快速、复杂且对开发不友好。语言的标准化发生了数十年。到2000年代，事情开始停滞。他们融合为两个阵营：Java/JVM和C/CLR。C++、Java、C＃都非常相似。
第三波浪潮：脚本语言特征：慢、不安全但对开发友好。脚本语言作为对上述语言的复杂性和痛苦的回应而应运而生。它们开发快速而松散，对开发人员友好，但缺乏性能和安全性。
第四波浪潮：恢复特征：快速、安全、对开发人员友好

Go恢复了早期语言的简单性和灵活性，增加了现代语言的安全性和开发友好性。Go以一种非常真实的方式复兴了许多伟大的想法，这些想法终于准备就绪。

Go给人的感觉就像是来自60年代，70年代，80年代，90年代，00s，10年代的语言……Steve Francia 2019

Go的设计哲学

原则1：进化不是革命(Evolution not revolution), 大多数思想都来自先前的思想大多数思想根本不是新事物进化不是革命：新语言应该巩固而不是发明新特性
原则2：等待良好的设计, No是暂时的，Yes是永远的(Waiting for Good design No is temporary, Yes is forever )。在Go的整个历史中，有很多这样的实例。通常的想法是，在设计语言时，不会出现“撤消(undo)”的情况。如果您今天说“No”，那么您明天总是可以说“Yes”，但是如果今天您说“Yes”，那么您将在很长一段时间或永远被它“困”住…。

如有疑问，请将其排除在外。- Joshua Bloch：关于设计的对话– 2002
原则3: 应该使一切都尽可能简单，但不要过于简单（Consensus driven design Everything should be made as simple as possible, but no simpler.）。-爱因斯坦

当我们三个人开始时，这纯粹是研究。…我们从一个想法开始，即我们三个人都必须针对该语言的每个特性进行讨论，因此，无论出于何种原因，都不会在该语言中放入多余的垃圾。- 肯·汤普森（Ken Thompson）访谈– 2011年，肯从Bell Labs学习了这种做法有两种构建软件设计的方法。一种方法是使其变得如此简单，以至于显然没有缺陷。另一种方法是使其变得如此复杂，以至于没有明显的缺陷。- 托尼·霍尔（Tony Hoare）皇帝的旧衣服-1981年，Go采取了第一种方法，而大多数其他语言都采用第二种方法。
快速迭代期待（Rapid iteration）并实现大规模改变最后一个原则是快速迭代的原则。

当您处于语言的设计阶段时，您将需要进行频繁且有时是巨大的更改。朝着这个期望前进，并围绕它建立您的流程。

bingoohuang commented 4 years ago

CentOS 上安装golang

curl -LO https://studygolang.com/dl/golang/go1.13.4.linux-amd64.tar.gz
sudo tar -C /usr/local/ -xzvf go1.13.4.linux-amd64.tar.gz
/usr/local/go/bin/go version
vi .bash_profile, export PATH=$PATH:/usr/local/go/bin
export http_proxy=http://192.168.33.1:9999; export https_proxy=http://192.168.33.1:9999;
go get -u -v golang.org/x/lint/golint
go get -u -v golang.org/x/lint/golint
curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s -- -b $(go env GOPATH)/bin v1.21.0

bingoohuang commented 4 years ago

如何有效地缩小go可执行文件大小

总结：use go build -ldflags="-s -w" ./... and then use UPX with --best --lzma binfile.

go build	go build -ldflags="-s -w"	upx hello	upx --brute hell
2.0M	1.6M (20% off)	1.6M（62% off，70%off total）	484K（21% off, 76% off total）

// hello.go
package main

import "fmt"

func main() {
    fmt.Printf("hello, world\n")
}

$ go version
go version go1.13.4 darwin/amd64
$ go build hello.go
$ ls -lh
-rwxr-xr-x  1 bingoo  staff   2.0M 11 12 22:51 hello
-rw-r--r--  1 bingoo  staff    86B 11 12 22:50 hello.go
$ go build -ldflags="-s -w" hello.go
$ ls -lh
-rwxr-xr-x  1 bingoo  staff   1.6M 11 12 22:51 hello
$ upx hello
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX 3.95        Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 26th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1665904 ->    630800   37.87%   macho/amd64   hello

Packed 1 file.
$ ls -lh
-rwxr-xr-x  1 bingoo  staff   616K 11 12 22:51 hello
$ go build -ldflags="-s -w" hello.go
$ upx --brute hello
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2018
UPX 3.95        Markus Oberhumer, Laszlo Molnar & John Reiser   Aug 26th 2018

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1665904 ->    495632   29.75%   macho/amd64   hello

Packed 1 file.
$ ls -lh
-rwxr-xr-x  1 bingoo  staff   484K 11 12 22:52 hello

Golang executables embed lots of symbolic information:

Debug symbols, implemented as DWARF symbols. These can be stripped at compilation time (command-line option -ldflags "-w") .
Classic symbols for each executable file format (PE/ELF/Mach-O). These can be stripped at compilation time (command-line option -ldflags "-s").
Go-specific metadata, including for example all functions’ entry points and names, and complete type information. These metadata cannot (easily) be stripped, because Golang runtime needs them: for example, functions’ information are needed to walk the stack for errors handling or for garbage collection, while types information serve for runtime type checks.

Shrinking Go Binaries

To reproduce my results, you can do the following:

go get -u -t -v github.com/taskcluster/taskcluster-lib-artifact-go
cd $GOPATH/src/github.com/taskcluster/taskcluster-lib-artifact-go
git checkout 6f133d8eb9ebc02cececa2af3d664c71a974e833
time (go build) && wc -c ./artifact
time (go build && strip ./artifact) && wc -c ./artifact
time (go build -ldflags="-s") && wc -c ./artifact
time (go build -ldflags="-w") && wc -c ./artifact
time (go build -ldflags="-s -w") && wc -c ./artifact
time (go build && upx -1 ./artifact) && wc -c ./artifact
time (go build && upx -9 ./artifact) && wc -c ./artifact
time (go build && strip ./artifact && upx -1 ./artifact) && wc -c ./artifact
time (go build && strip ./artifact && upx --brute ./artifact) && wc -c ./artifact
time (go build && strip ./artifact && upx --ultra-brute ./artifact) && wc -c ./artifact
time (go build && strip && upx -9 ./artifact) && wc -c ./artifact

linux中的strip命令简介------给文件脱衣服

man strip: strip removes or modifies the symbol table attached to the output of the assembler and link editor. This is useful to save space after a program has been debugged and to limit dynamically bound symbols.

strip用于脱掉文件的衣服，文件会变小，其中的符号信息会失去。那这个strip有什么用呢？很有用的！原来的a.out比较大，可以执行。在strip之后，文件变小了，仍然可以执行，这就就节省了很多空间。 strip不仅仅可以针对可执行文件，还能针对目标文件和动态库等。在实际的开发中，经常需要对动态库.so进行strip操作，减少占地空间。而在调试的时候（比如用addr2line），就需要符号了。因此，通常的做法是： strip前的库用来调试， strip后的库用来实际发布，他们两者有对应关系。一旦发布的strip后的库出了问题，就可以找对应的未strip的库来定位。

bingoohuang commented 4 years ago

出现问题，依赖高版本的glibc包，目标机器centos6.9，没有，报错。

$ ./rigaga
./rigaga: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./rigaga)
$ ll /lib64/libc.so.6
lrwxrwxrwx. 1 root root 12 6月   8 2018 /lib64/libc.so.6 -> libc-2.12.so

golang 独立打包（不依赖系统动态链接库）

$ go build -o sysinfo cmd/sysinfo/main.go
$ ldd sysinfo
    linux-vdso.so.1 =>  (0x00007ffdff179000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b1cff7fc000)
    libc.so.6 => /lib64/libc.so.6 (0x00002b1cffa18000)
    /lib64/ld-linux-x86-64.so.2 (0x00002b1cff5d8000)
[footstone@fs01-192-168-126-182 sysinfo]$ go build -o sysinfo  -ldflags '-linkmode "external" -extldflags "-static"' cmd/sysinfo/main.go
# command-line-arguments
/tmp/go-link-033563708/000002.o: In function `mygetgrouplist':
/workdir/go/src/os/user/getgrouplist_unix.go:16: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetgrgid_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetgrnam_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetpwnam_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetpwuid_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000008.o: In function `_cgo_7e1b3c2abc8d_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
$ ldd sysinfo
    not a dynamic executable

Site:

Go build with another glibc

Before doing go build Set CGO_LDFLAGS

Dynamic: export CGO_LDFLAGS="-Xlinker -rpath=/path/to/another_glibc/lib"

Static: export CGO_LDFLAGS="-Xlinker -rpath=/path/to/another_glibc/lib -static"

CGO_LDFLAGS lets you set GCC-like ld flags for Go.

bingoohuang commented 4 years ago

如果汽车像电脑一样发展的话，会是什么样子？If cars developed at the same pace as computers, they’d be this fast

汽车省不省油可以通过计算MPG来大概得到, 所谓的MPG就是 Miles Per Gallon 也就是每加伦油车能开的大概距离, 这个数值是越高越好. 一般来说, 在城市, 车辆MPG大概在20到40之间, 而在郊区或乡村, MPG能到50以上.
英制的马力 ( HP ) 是英国物理学家瓦特为了测定新制造出来的蒸汽机的功率而定义的单位。是在他家附近的酒厂主有一匹壮马，瓦特改良蒸汽机后想拿他发明的 " 蒸汽马 " 和真马较量一番，于是就和酒厂主用那匹马来拉吊在定滑轮上的水桶，马在一分钟里，把 1000 磅重（约合 4540 牛顿）的重物提高 33 英尺（约 10 米）。瓦特就把这匹马的功率规定为 1 马力，而 HP 这个单位也就是 Horse Power 的略称，后来就发展为了汽车功率的单位。

bingoohuang commented 4 years ago

Go Anti-Patterns : os.IsExist(err) != !os.IsNotExist(err) , [原文](https://stefanxo.com/go-anti-patterns-os-isexisterr-os-isnotexisterr/）

Example 1: We want to create a symlink, but the target path exists already :

if _, err := os.Symlink("/old/path", "/path/to/whatever"); os.IsExist(err) {
  // error happened, can't Symlink
  // can't create symlink because /path/to/whatever already exists
}

Example 2: We want to stat a file, but it doesn't exist :

if _, err := os.Stat("/path/to/whatever"); os.IsNotExist(err) {
  // error happened, can't Stat!
  // /path/to/whatever does not exist
}

Anti-Pattern: And here comes the big BUT :

// Anti-pattern : We want to stat a file, and continue if it exists :
if _, err := os.Stat("/file/that/exists"); os.IsExist(err) {
  // will never trigger!
  // why? because os.Stat runs normally if file exists.
  // it's expected behaviour for os.Stat, so it doesn't throw an error
  // os.IsExist() does not receive an error ( it's nil ), so it can't tell you
  // if the error message was "file not found"
}

Instead we want to stat a file, and continue if it exists :

_, err := os.Stat("/file/that/exists");
if err != nil {
    if os.IsNotExist(err) {
        // file does not exist, do something
    } else {
        // more serious errors
    }
}
// file exists.. continue with code here, or in else statement or specify if err == nil { // do something }

os.IsExist(err) is good for cases when you expect the file to not exist yet, but the file actually exists 👍

os.Symlink("/path/that/exists", "/path/to/symlink/target")
// os.IsExist(err) will trigger when target exists already
os.Mkdir(target)
// os.IsExist(err) will trigger because target path already exists
os.OpenFile(target, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0600)
// os.IsExist(err) will trigger when target exists because O_EXCL means that file should not exist yet

os.IsNotExist(err) is good for more common cases where you expect the file to exists, but it actually doesn't exist :

os.Chdir()
os.Stat()
os.Open()
os.OpenFile() // without os.O_EXCL
os.Chmod()
os.Chown()
os.Close()
os.Read()
os.ReadAt()
os.ReadDir()
os.Readdirnames()
os.Seek()
os.Truncate()
os.Write()
os.WriteAt()
os.WriteString()

Channel 应用模式

Channel的特殊情况汇总

~	ni	not empty	empty	full	not full
Receive	block	value	block	value	value
Send	block	write value	write value	block	write value
close	panic	closed, drained read, return zero value	closed, return zero value	closed, drained read, return zero value	closed, drained read, return zero value

Lock/TryLock模式

or 信号模式

从多个channel读取一个信号, 一旦读取到一个信号，则不再读取。

比如向多个服务器发送相同的http request,每个请求的结果放在单独的一个channel中，只要其中一个服务器返回结果，则其它请求就被忽略。

or channel by goroutine

or_channel

最简单的方式就是为每个channel启动一个goroutine, 每个goroutine读取自己负责的channel，一旦读取到一个信号，就关闭返回的channel。显然，为每个channel启动一个goroutine太浪费了，虽然goroutine是一种轻量级的实现，但是如果数量巨大的情况下也会导致资源的大量占用以及调度上的性能低下。

or channel go(递归)

or_channel_go 基于递归的方式实现，使用依次递归的方式

or channel_rec (递归)

or_channel_rec 基于递归的方式实现, 使用分而治之的方式

or channel reflect

or_channel_reclect 基于反射的方式

or_done_channel模式

与上面的or 信号模式不同， or done channel模式是从一个channel中读取数据，只有当channel被关闭，或者done 信号channel被关闭的时候，读取操作才退出。

or_done_channel

如果将done这个信号channel换成 context,则可以依靠 context.WithCancel 来cancel读取，和这个模式类似。

flat 模式

flat 将多个channels平展成一个channels。与Fan In不同的是，输入的channels是从一个channel中读取出来的，而Fan In模式中的channels是一个channel slice。

map/reduce 模式

mapreduce

Fan In 扇入模式

将多个channel合并成一个channel

fanIn

Fan Out 扇出模式

将一个Channel分成多个Channel。有两种情况，一种是每个channel都包含同样的数据(复制模式)，另一种将原数据均匀分布到各输出channel中(分布模式)

复制模式

fanOut
fanOutReflect

分布模式

fanOut
fanOutReflect

参考

bingoohuang commented 4 years ago

Go调度器

                    +-------+       +-------+      
                    |  KSE  |       |  KSE  |          
                    +-------+       +-------+      
                        |               |                       内核空间
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -        
                        |               |                       用户空间
                    +-------+       +-------+
                    |   M   |       |   M   |
                    +-------+       +-------+
                  |          |         |          |
              +------+   +------+   +------+   +------+            
              |   P  |   |   P  |   |   P  |   |   P  |
              +------+   +------+   +------+   +------+   
           |     |     |     |     |     |     |     |     | 
         +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ 
         | G | | G | | G | | G | | G | | G | | G | | G | | G | 
         +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+

GPM模型

Thanks

bingoohuang commented 4 years ago

How to point Go module dependency in go.mod to a latest commit in a repo?

Just 'go get' at the commit hash you want:

go get github.com/someone/some_module@af044c0995fe

'go get' will correctly update the dependency files (go.mod, go.sum).

More information

bingoohuang commented 4 years ago

Erlang的作者Armstrong在其博士论文中证明，单个机器自身，由于对其硬件出错的不可挽回，本质上是不可靠的。但独立机器构成的网络，可以互相监督和备份，在容错系统软件的支持下，完全有能力大大提高整体的可靠性到九个9，也就是理论上几乎完全可靠。

由于有了go func，除了在main大厅外，Go还可以轻易的安排非常多的小房间，和main大厅里的同时func。这种互不打扰的群体func行为史称并发。 runtime就是龟婆，提供func执行的环境。defer是延迟的意思好似伟哥，她把本来要马上发生的f()行为推迟到最后发生。 panic就像二奶一样见光死。

胡文Go.ogle-Fango

bingoohuang commented 4 years ago

优化点在于 for 循环里不要使用 select + time.After 的组合。

time.After 在压测的时候，我们发现内存占用很高，于是使用 Go Tool PProf 分析 Golang 函数内存申请情况，发现有不断创建 time.After 定时器的问题，定位到是心跳协程里面。
JSON 编解码

开始我们使用官方的 JSON 编解码工具，但由于对性能方面的追求，改为使用滴滴开源的 Json-iterator，使在兼容原生 Golang 的 JSON 编解码工具的同时，效率上有比较明显的提升。以下是压测对比的参考图：
日志模块在起初调研的时候基于性能考虑，确定使用 Uber 开源的 ZAP 库，而且满足业务日志记录的要求。日志库选型很重要，选不好也是影响系统性能和稳定性的。ZAP 的优点包括：

显示代码行号这个需求，ZAP 支持而 Logrus 不支持，这个属于提效的。行号展示对于定位问题很重要。 ZAP 相对于 Logrus 更为高效，体现在写 JSON 格式日志时，没有使用反射，而是用内建的 json encoder，通过明确的类型调用，直接拼接字符串，最小化性能开销。小坑：每天写一个日志文件的功能，目前 ZAP 不支持，需要自己写代码支持，或者请求系统部支持。

Go 称为云原生时代的母语。「云原生时代，是开发者最好的时代」，在这股浪潮下，我们越早走进 Go，就可能越早在这个新时代抢占关键赛道。

bingoohuang commented 4 years ago

不错的文章Mark, 阿里云高级技术专家探讨 Go 的错误处理

哪些可以 Recover
1. recover 只有在 defer 函数中才有用，在 defer 的函数调用的函数中 recover 不起作用
2. Slice 越界
3. nil 指针被引用的例子
4. 除零
5. 写关闭的 chan 的例子
Recover 最佳实践
1. 一般 recover 后会判断是否 err，有可能需要处理特殊的 error，一般也需要打印日志或者告警
2. 对于库如果需要启动 goroutine，如何 recover 呢？
  1. 如果不可能出现 panic，可以不用 recover，比如 tls.go 中的一个 goroutine：errChannel <- conn.Handshake() ；
  2. 如果可能出现 panic，也比较明确的可以 recover，可以调用用户回调，或者让用户设置 logger，比如 http/server.go 处理请求的 goroutine：if err := recover(); err != nil && err != ErrAbortHandler { ；
  3. 如果完全不知道如何处理 recover，比如一个 cache 库，丢弃数据可能会造成问题，那么就应该由用户来启动 goroutine，返回异常数据和错误，用户决定如何 recover、如何重试；
  4. 如果完全知道如何 recover，比如忽略 panic 继续跑，或者能使用 logger 打印日志，那就按照正常的 panic-recover 逻辑处理。
哪些不能 Recover，包括（不限于）：
1. Go1.2 引入了能使用的最多线程数限制ThreadLimit，如果超过了就 panic，这个 panic 是无法 recover 的。
  
  默认是 1 万个物理线程，我们可以调用 runtime 的debug.SetMaxThreads 设置最大线程数。 goroutine 启动时，并不总是新开系统线程，只有当目前所有的物理线程都阻塞在系统调用、cgo 调用，或者显示有调用 runtime.LockOSThread 时。 GOMAXPROCS 只是设置 user-level 并行执行的线程数，也就是真正执行的线程数。实际上如果物理线程阻塞在 system calls，会开启更多的物理线程。
2. Concurrent Map Writers，竞争条件，同时写 map，参考下面的例子。推荐使用标准库的 sync.Map 解决这个问题。

Errors in Go，Go 中正确的错误处理，应该是这个例子 Good: CopyFile，虽然啰嗦繁琐不简洁：

package main

import (
  "fmt"
  "io"
  "os"
)

func CopyFile(src, dst string) error {
  r, err := os.Open(src)
  if err != nil {
    return fmt.Errorf("copy %s %s: %v", src, dst, err)
  }
  defer r.Close()

  w, err := os.Create(dst)
  if err != nil {
    return fmt.Errorf("copy %s %s: %v", src, dst, err)
  }

  if _, err := io.Copy(w, r); err != nil {
    w.Close()
    os.Remove(dst)
    return fmt.Errorf("copy %s %s: %v", src, dst, err)
  }

  if err := w.Close(); err != nil {
    os.Remove(dst)
    return fmt.Errorf("copy %s %s: %v", src, dst, err)
  }
  return nil
}

func main() {
  fmt.Println(CopyFile("src.txt", "dst.txt"))
}

总结下 Go 的 error，错误处理应该注意以下几点：
- 凡是有返回错误码的函数，必须显式的处理错误，如果要忽略错误，也应该显式的忽略和写注释；
- 错误必须带丰富的错误信息，比如堆栈、发生错误时的参数、调用链给的描述等等。特别要强调变量，我看过太多日志描述了一对常量，比如 "Verify the nonce, timestamp and token of specified appid failed"，而这个消息一般会提到工单中，然后就是再问用户，哪个 session 或 request 甚至时间点？这么一大堆常量有啥用呢，关键是变量呐；尽量避免重复的信息，提高错误处理的开发体验，糟糕的体验会导致无效的错误处理代码，比如拷贝和漏掉关键信息；
- 分离错误和日志，发生错误时返回带完整信息的错误，在调用的顶层决定是将错误用日志打印，还是发送到监控系统，还是转换错误，或者忽略。

Best Practice

推荐用github.com/pkg/errors 这个错误处理的库，基本上是够用的，参考 Refine: CopyFile，可以看到 CopyFile 中低级重复的代码已经比较少了：

    package main

    import (
      "fmt"
      "github.com/pkg/errors"
      "io"
      "os"
    )

    func CopyFile(src, dst string) error {
      r, err := os.Open(src)
      if err != nil {
        return errors.Wrap(err, "open source")
      }
      defer r.Close()

      w, err := os.Create(dst)
      if err != nil {
        return errors.Wrap(err, "create dest")
      }

      nn, err := io.Copy(w, r)
      if err != nil {
        w.Close()
        os.Remove(dst)
        return errors.Wrap(err, "copy body")
      }

      if err := w.Close(); err != nil {
        os.Remove(dst)
        return errors.Wrapf(err, "close dest, nn=%v", nn)
      }

      return nil
    }

    func LoadSystem() error {
      src, dst := "src.txt", "dst.txt"
      if err := CopyFile(src, dst); err != nil {
        return errors.WithMessage(err, fmt.Sprintf("load src=%v, dst=%v", src, dst))
      }

      // Do other jobs.

      return nil
    }

    func main() {
      if err := LoadSystem(); err != nil {
        fmt.Printf("err %+v\n", err)
      }
    }

日志的理解
1. 日志是给人看的，是用来查问题的。出现问题后根据某些条件，去查不同进程或服务的日志。日志的关键是不能漏掉信息，漏了关键日志，可能就断了关键的线索；
2. 日志必须要被关联起来，上下文的日志比单个日志更重要。长连接需要根据会话关联日志；不同业务模型有不同的上下文，比如服务器管理把服务器作为关键信息，查询这个服务器的相关日志；全链路跨机器和服务的日志跟踪，需要定义可追踪的逻辑 ID；常见的上下文包括：
  1. 如果是短连接，一条日志就能描述，那可能要将多个服务的日志关联起来，将全链路的日志作为上下文；
  2. 如果是长连接，一般长连接一定会有定时信息，比如每隔 5 秒输出这个链接的码率和包数，这样每个链接就无法使用一条日志描述了，链接本身就是一个上下文；
  3. 进程内的逻辑上下文，比如代理的上下游就是一个上下文，合并回源，故障上下文，客户端重试等。
3. 海量日志是给机器看的，是结构化的，能主动报告问题，能从日志中分析潜在的问题。日志的关键是要被不同消费者消费，要输出不同主题的日志，不同的粒度的日志。日志可以用于排查问题，可以用于告警，可以用于分析业务情况。
4. 一些最佳实践：
  1. 用 Context 的 WithValue 来将上下文相关的 ID 保存，在打印日志时将 ID 取出来；
  2. 如果有业务特征，比如可以取 SessionID 的 hash 的前 8 个字符形成 ID，虽然容易碰撞，但是在一定范围内不容易碰撞；
  3. 可以变成 json 格式的日志，这样可以将 level、id、tag、file、err 都变成可以程序分析的数据，送到 SLS 中处理；
  4. 对于切割和轮转，推荐使用lumberjack 这个库，程序的 logger 只要提供 SetOutput(io.Writer) 将日志送给它处理就可以了。

bingoohuang commented 4 years ago

GO阅读:如何写出优雅的 Go 语言代码

在这篇文章中从三个方面分别介绍了如何写优雅的 Go 语言代码，作者尽可能地给出了最容易操作和最有效的方法：

代码规范：使用辅助工具帮助我们在每次提交 PR 时自动化地对代码进行检查，减少工程师人工审查的工作量；
最佳实践
- 目录结构：遵循 Go 语言社区中被广泛达成共识的目录结构，减少项目的沟通成本；
- 模块拆分：按照职责对不同的模块进行拆分，Go 语言的项目中也不应该出现 model、controller 这种违反语言顶层设计思路的包名；
- 显示与隐式：尽可能地消灭项目中的 init 函数，保证显式地进行方法的调用以及错误的处理；
- 面向接口：面向接口是 Go 语言鼓励的开发方式，也能够为我们写单元测试提供方便，我们应该遵循固定的模式对外提供功能；
  1. - 使用大写的 Service 对外暴露方法；
  2. - 使用小写的 service 实现接口中定义的方法；
  3. - 通过 func NewService(...) (Service, error) 函数初始化 Service 接口；
单元测试：保证项目工程质量的最有效办法；
- 可测试：意味着面向接口编程以及减少单个函数中包含的逻辑，使用『小方法』；
- 组织方式：使用 Go 语言默认的 Test 框架、开源的 suite 或者 BDD 的风格对单元测试进行合理组织；
- Mock 方法：四种不同的单元测试 Mock 方法；
  1. gomock：最标准的也是最被鼓励的方式；
  2. sqlmock：处理依赖的数据库；
  3. httpmock：处理依赖的 HTTP 请求；
  4. monkey：万能的方法，但是只在万不得已时使用，类似的代码写起来非常冗长而且不直观；
- 断言：使用社区的 testify 快速验证方法的返回值；

想要写出优雅的代码本身就不是一件容易的事情，它需要我们不断地对自己的知识体系进行更新和优化，推倒之前的经验并对项目持续进行完善和重构，而只有真正经过思考和设计的代码才能够经过时间的检验（代码是需要不断重构的），随意堆砌代码的行为是不能鼓励也不应该发生的，每一行代码都应该按照最高的标准去设计和开发，这是我们保证工程质量的唯一方法。

作者也一直在努力学习如何写出更加优雅的代码，写出好的代码真的不是一件容易的事情，作者也希望能通过这篇文章帮助使用工程师写出更有 Go 语言风格的项目。

bingoohuang commented 4 years ago

GOLANG私服

图片来自nexus帮助

nexus 3.17.0 做为golang 的包管理工具

# docker-compose
version: "3"
services: 
  nexus:
    image: sonatype/nexus3:3.17.0
    ports: 
    - "80:8081"
    volumes: 
    - "./nexus-data:/nexus-data"
  athens:
    image: gomods/athens:latest
    ports: 
    - "3000:3000"

配置go proxy

bingoohuang commented 4 years ago

从go可执行程序中，查看go的编译版本

$ strings weed  | grep "linux.amd64" | sed -n '2,10 p'
/home/travis/.gimme/versions/go1.12.6.linux.amd64
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/container/heap/heap.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/container/ring/ring.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/log/syslog/syslog_unix.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/log/syslog/syslog.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/base32/base32.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/gob/encoder.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/gob/encode.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/gob/enc_helpers.go

Get golang version from compiled binary

bingoohuang commented 4 years ago

An Overview of Go's Tooling

Posted on: 15th April 2019
Filed under: golang tutorial

Occasionally I get asked “why do you like using Go?” And one of the things I often mention is the thoughtful tooling that exists alongside the language as part of the go command. There are some tools that I use everyday — like go fmt and go build — and others like go tool pprof that I only use to help solve a specific issue. But in all cases I appreciate the fact that they make managing and maintaining my projects easier.

In this post I hope to provide a little background and context about the tools I find most useful, and importantly, explain how they can fit into the workflow of a typical project. I hope it'll give you a good start if you're new to Go.

Or if you've been working with Go for a while, and that stuff's not applicable to you, hopefully you'll still discover a command or flag that you didn't know existed before : )

The information in this post is written for Go 1.12 and assumes that you're working on a project which has modules enabled.

Installing Tooling
Viewing Environment Information
Development
1. Running Code
2. Fetching Dependencies
3. Refactoring Code
4. Viewing Go Documentation
Testing
1. Running Tests
2. Profiling Test Coverage
3. Stress Testing
4. Testing all Dependencies
Pre-Commit Checks
1. Formatting Code
2. Performing Static Analysis
3. Linting Code
4. Tidying and Verifying your Dependencies
Build and Deployment
1. Building an Executable
2. Cross-Compilation
3. Using Compiler and Linker Flags
Diagnosing Problems and Making Optimizations
1. Running and Comparing Benchmarks
2. Profiling and Tracing
3. Checking for Race Conditions
Managing Dependencies
Upgrading to a New Go Release
Reporting Bugs
Cheatsheet

Installing Tooling

In this post I'll mainly be focusing on tools that are a part of the go command. But there are a few I'll be mentioning which aren't part of the standard Go 1.12 release.

To install these while using Go 1.12 you'll first need to make sure that you're outside of a module-enabled directory (I usually just change into /tmp). Then you can use the GO111MODULE=on go get command to install the tool. For example:

$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/cmd/stress

This will download the relevant package and dependencies, build the executable and add it to your GOBIN directory. If you haven't explicitly set a GOBIN directory, then the executable will be added to your GOPATH/bin folder. Either way, you should make sure that the appropriate directory is on your system path.

Note: This process is a bit clunky and will hopefully improve in future versions of Go. Issue 30515 is tracking the discussion about this.

Viewing Environment Information

You can use the go env tool to display information about your current Go operating environment. This can be particularly useful if you're working on an unfamiliar machine.

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/alex/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build245740092=/tmp/go-build -gno-record-gcc-switches"

If there are specific values that you're interested in, you can pass them as arguments to go env. For example:

$ go env GOPATH GOOS GOARCH
/home/alex/go
linux
amd64

To show documentation for all go env variables and values you can run:

$ go help environment

Development

Running Code

During development the go run tool is a convenient way to try out your code. It's essentially a shortcut that compiles your code, creates an executable binary in your /tmp directory, and then runs this binary in one step.

$ go run .          # Run the package in the current directory
$ go run ./cmd/foo  # Run the package in the ./cmd/foo directory

Note: As of Go 1.11 you can pass the path of a package to go run, like we have above. This means that you no longer have to use workarounds like go run *.go wildcard expansion to run multiple files. I like this improvement a lot!

Fetching Dependencies

Assuming that you've got modules enabled, when you use go run (or go test or go build for that matter) any external dependencies will automatically (and recursively) be downloaded to fulfill the import statements in your code. By default the latest tagged release of the dependency will be downloaded, or if no tagged releases are available, then the dependency at the latest commit.

If you know in advance that you need a specific version of a dependency (instead of the one that Go would fetch by default) you can use go get with the relevant version number or commit hash. For example:

$ go get github.com/foo/bar@v1.2.3
$ go get github.com/foo/bar@8e1b8d3

If the dependency being fetched has a go.mod file, then its dependencies won't be listed in your go.mod file. In contrast, if the dependency you're downloading doesn't have a go.mod file, then it's dependencies will be listed in your go.mod file with an // indirect comment next to them.

So that means your go.mod file doesn't necessarily show all the dependencies for your project in one place. Instead, you can view them all using the go list tool like so:

$ go list -m all

Sometimes you might wonder why is that a dependency? You can answer this with the go mod why command, which will show you the shortest path from a package in your main module to a given dependency. For example:

$ go mod why -m golang.org/x/sys
# golang.org/x/sys
github.com/alexedwards/argon2id
golang.org/x/crypto/argon2
golang.org/x/sys/cpu

Note: The go mod why command will return an answer for most, but not all, dependencies. Issue 27900 is tracking this.

If you're interested in analyzing or visualizing the dependencies for your application, then you might want to also check out the go mod graph tool. There's a great tutorial and example code for generating visualizations here.

Lastly, downloaded dependencies are stored in the module cache located at GOPATH/pkg/mod. If you ever need to clear the module cache you can use the go clean tool. But be aware: this will remove the downloaded dependencies for all projects on your machine.

$ go clean -modcache

Refactoring Code

Chances are you're probably familiar with using the gofmt tool to automatically format your code. But it also supports rewrite rules that you can use to help refactor your code. I'll demonstrate.

Let's say that you have the following code and you want to change the foo variable to Foo so it is exported.

var foo int

func bar() {
    foo = 1
    fmt.Println("foo")
}

To do this you can use gofmt with the -r flag to implement a rewrite rule, the -d flag to display a diff of the changes, and the -w flag to make the changes in place, like so:

$ gofmt -d -w -r 'foo -> Foo' .
-var foo int
+var Foo int

 func bar() {
-   foo = 1
+   Foo = 1
    fmt.Println("foo")
 }

Notice how this is smarter than a find-and-replace? The foo variable has been changed, but the "foo" string in the fmt.Println() statement has been left unchanged. Another thing to note is that the gofmt command works recursively, so the above command will run on all *.go files in your current directory and subdirectories.

If you want to use this functionality, I recommend running rewrite rules without the -w flag first, and checking the diff first to make sure that the changes to the code are what you expect.

Let's take a look at a slightly more complicated example. Say you want to update your code to use the new Go 1.12 strings.ReplaceAll() function instead of strings.Replace(). To make this change you can run:

$ gofmt -w -r 'strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)' .

In rewrite rules, single lowercase characters act as wildcards matching arbitrary expressions, and those expressions will be substituted-in in the replacement.

Viewing Go Documentation

You can view documentation for the standard library packages via your terminal using the go doc tool. I often use this during development to quickly check something — like the name or signature of a specific function. I find it faster than navigating the web-based documentation and it's always available offline too.

$ go doc strings            # View simplified documentation for the strings package
$ go doc -all strings       # View full documentation for the strings package
$ go doc strings.Replace    # View documentation for the strings.Replace function
$ go doc sql.DB             # View documentation for the database/sql.DB type
$ go doc sql.DB.Query       # View documentation for the database/sql.DB.Query method

You can also include the -src flag to display the relevant Go source code. For example:

$ go doc -src strings.Replace   # View the source code for the strings.Replace function

Testing

Running Tests

You can use the go test tool to run tests in your project like so:

$ go test .          # Run all tests in the current directory
$ go test ./...      # Run all tests in the current directory and sub-directories
$ go test ./foo/bar  # Run all tests in the ./foo/bar directory

Typically I run my tests with Go's race detector enabled, which can help pick up some of the data races that might occur in real-life usage. Like so:

$ go test -race ./...

It's important to note that enabling the race detector will increase the overall running time of your tests. So if you're running tests very frequently part of a TDD workflow, you might prefer to save using this for a pre-commit test run only.

Since 1.10, Go caches test results at the package-level. If a package hasn't changed between test runs — and you're using the same, cachable, flags for go test — then the cached test result will be displayed with a "(cached)" next to it. This is hugely helpful in speeding up the test runtime for large codebases. If you want force your tests to run in full (and avoid the cache) you can use the -count=1 flag, or clear all cached test results by using the go clean tool.

$ go test -count=1 ./...    # Bypass the test cache when running tests
$ go clean -testcache       # Delete all cached test results

Note: Cached test results are stored alongside cached build results in your GOCACHE directory. Check go env GOCACHE if you're not sure where this is on your machine.

You can limit go test to running specific tests (and sub-tests) by using the -run flag. This accepts a regular expression, and only tests which have names that match the regular expression will be run. I like to combine this with the -v flag to enable verbose mode, so the names of running tests and sub-tests are displayed. It's a useful way to make sure that I haven't screwed up the regexp and that the tests I expect are actually being run!

$ go test -v -run=^TestFooBar$ .          # Run the test with the exact name TestFooBar
$ go test -v -run=^TestFoo .              # Run tests whose names start with TestFoo
$ go test -v -run=^TestFooBar$/^Baz$ .    # Run the Baz subtest of the TestFooBar test only

A couple more flags that it's good to be aware of are -short (which you can use to skip long-running tests) and -failfast (which will stop running further tests after the first failure). Note that -failfast will prevent test results from being cached.

$ go test -short ./...      # Skip long running tests
$ go test -failfast ./...   # Don't run further tests after a failure.

Profiling Test Coverage

You can enable coverage analysis when running tests by using the -cover flag. This will display the percentage of code covered by the tests in the output for each package, similar to this:

$ go test -cover ./...
ok      github.com/alexedwards/argon2id 0.467s  coverage: 78.6% of statements

You can also generate a coverage profile using the -coverprofile flag and view it in your web browser by using the go tool cover -html command like so:

$ go test -coverprofile=/tmp/profile.out ./...
$ go tool cover -html=/tmp/profile.out

This will gives you a navigable listing of all the test files, with code covered by the tests displayed in green, and uncovered code in red.

If you want you can go a step further and set the -covermode=count flag to make the coverage profile record the exact number of times that each statement is executed during the tests.

$ go test -covermode=count -coverprofile=/tmp/profile.out ./...
$ go tool cover -html=/tmp/profile.out

When viewed in the browser, statements which are executed more frequently are shown in a more saturated shade of green, similar to this:

Note: If you’re using the t.Parallel() command in any of your tests, then you should use the flag -covermode=atomic instead of -covermode=count instead to ensure an accurate count.

Lastly, if you don't have a web browser available to view a coverage profile, you can see a breakdown of test coverage by function/method in your terminal with the command:

$ go tool cover -func=/tmp/profile.out
github.com/alexedwards/argon2id/argon2id.go:77:     CreateHash      87.5%
github.com/alexedwards/argon2id/argon2id.go:96:     ComparePasswordAndHash  85.7%
...

Stress Testing

You can use the go test -count command to run a test multiple times in succession, which can be useful if you want to check for sporadic or intermittent failures. For example:

$ go test -run=^TestFooBar$ -count=500 .

In this example, the TestFooBar test will be repeated 500 times in a row. But it's important to note that the test will be repeated in serial — even if it contains a t.Parallel() instruction. So if your test is doing something relatively slow, like making a round trip to a database, hard disk or the internet, running a large number of tests can take quite a long time.

In that case you might want to use the stress tool to repeat the same test multiple times in parallel instead. You can install it like so:

$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/cmd/stress

To use the stress tool, you'll first need to compile a test binary for the specific package you want to test. You can do using the go test -c command. For example, to create a test binary for the package in your current directory:

$ go test -c -o=/tmp/foo.test .

In this example, the test binary will be outputted to /tmp/foo.test. You can then use the stress tool to execute a specific test in the test binary like so:

$ stress -p=4 /tmp/foo.test -test.run=^TestFooBar$
60 runs so far, 0 failures
120 runs so far, 0 failures
...

Note: In the example above I've used the -p flag to restrict the number of parallel processes used by stress to 4. Without this flag, the tool will default to using a number of processes equal to runtime.NumCPU().

Testing all Dependencies

Before you build an executable for release or deployment, or distribute your code publicly, you may want to run the go test all command:

$ go test all

This will run tests on all packages in your module and all dependencies — include testing test dependencies and the necessary standard library packages — and it can help validate that the exact versions of the dependencies being used are compatible with each other. This can take quite a long time to run, but the results cache well so any subsequent tests should be faster in the future. If you want, you could also use go test -short all to skip any long-running tests.

Pre-Commit Checks

Formatting Code

Go provides two tools to automatically format your code according to the Go conventions: gofmt and go fmt. Using these helps keep your code consistent across your files and projects, and — if you use them before committing code — helps reduce noise when examining a diff between file versions.

I like to use the gofmt tool with the following flags:

$ gofmt -w -s -d foo.go  # Format the foo.go file
$ gofmt -w -s -d .       # Recursively format all files in the current directory and sub-directories

In these commands, the -w flag instructs the tool to rewrite files in place, the -s instructs the tool to apply simplifications to the code where possible, and the -d flag instructs the tool to output diffs of the changes (because I'm curious to see what is changed). If you want to only display the names of changed files, instead of diffs, you can swap this for the -l flag instead.

Note: The gofmt command works recursively. If you pass it a directory like . or ./cmd/foo it'll format all .go files under the directory.

The other formatting tool — go fmt — tool is a wrapper which essentially calls gofmt -l -w on a specified file or directory. You can use it like this:

$ go fmt ./...

Performing Static Analysis

The go vet tool carries out static analysis of your code and warns you of things which might be wrong with your code but wouldn't be picked up by the compiler. Issues like unreachable code, unnecessary assignments and badly-formed build tags. You can use it like so:

$ go vet foo.go     # Vet the foo.go file
$ go vet .          # Vet all files in the current directory
$ go vet ./...      # Vet all files in the current directory and sub-directories
$ go vet ./foo/bar  # Vet all files in the ./foo/bar directory

Behind the scenes, go vet runs a bunch of different analyzers which are listed here and you can disable specific ones on a case-by-case basis. For example to disable the composite analyzer you can use:

$ go vet -composites=false ./...

There are a couple of experimental analyzers in golang.org/x/tools which you might want to try: nilness (which checks for redundant or impossible nil comparisons) and shadow (which check for possible unintended shadowing of variables). If you want to use these, you'll need to install and run them separately. For example, to install nilness you would run:

$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness

And you can then use it like so:

$ go vet -vettool=$(which nilness) ./...

Note: when the -vettool flag is used it will only run the specified analyzer — all the other go vet analyzers won't be run.

As a side note, since Go 1.10 the go test tool automatically executes a small, high-confidence, subset of the go vet checks before running any tests. You can turn this behavior off when running tests like so:

$ go test -vet=off ./...

Linting Code

You can use the golint tool to identify style mistakes in your code. Unlikego vet`, this isn't concerned with correctness of the code, but helps you to align your code with the style conventions in Effective Go and the Go CodeReviewComments.

It's not part of the standard library, so you'll need to install it like so:

$ cd /tmp
$ GO111MODULE=on go get golang.org/x/lint/golint

You can then run it as follows:

$ golint foo.go     # Lint the foo.go file
$ golint .          # Lint all files in the current directory
$ golint ./...      # Lint all files in the current directory and sub-directories
$ golint ./foo/bar  # Lint all files in the ./foo/bar directory

Tidying and Verifying your Dependencies

Before you commit any changes to your code I recommend running the following two commands to tidy and verify your dependencies:

$ go mod tidy
$ go mod verify

The go mod tidy command will prune any unused dependencies from your go.mod and go.sum files, and update the files to include dependencies for all possible build tags/OS/architecture combinations (note: go run, go test, go build etc are ‘lazy' and will only fetch packages needed for the current build tags/OS/architecture). Running this before each commit will make it easier to determine which of your code changes were responsible for adding or removing which dependencies when looking at the version control history.

I also recommend using the go mod verify command to check that the dependencies on your computer haven't accidentally (or purposely) been changed since they were downloaded and that they match the cryptographic hashes in your go.sum file. Running this helps ensure that the dependencies being used are the exact ones that you expect, and any build for that commit will be reproducible at a later point.

Build and Deployment

Building an Executable

To compile a main package and create an executable binary you can use the go build tool. Typically I use it in conjunction with the -o flag, which let's you explicitly set the output directory and name of the binary like so:

$ go build -o=/tmp/foo .            # Compile the package in the current directory
$ go build -o=/tmp/foo ./cmd/foo    # Compile the package in the ./cmd/foo directory

In these examples, go build will compile the specified package (and any dependent packages), then invoke the linker to generate an executable binary, and output this to /tmp/foo.

It's important to note that, as of Go 1.10, the go build tool caches build output in the build cache. This cached output will be reused again in future builds where appropriate, which can significantly speed up the overall build time. This new caching behavior means that the old maxim of “prefer go install to go build to improve caching” no longer applies.

If you're not sure where your build cache is, you can check by running the go env GOCACHE command:

$ go env GOCACHE
/home/alex/.cache/go-build

Using the build cache comes with one important caveat — it does not detect changes to C libraries imported with cgo. So if your code imports a C library via cgo and you've made changes to it since the last build, you'll need to use the -a flag which forces all packages to be rebuilt. Alternatively, you could use go clean to purge the cache:

$ go build -a -o=/tmp/foo .     # Force all packages to be rebuilt
$ go clean -cache               # Remove everything from the build cache

Note: Running go clean -cache will delete cached test results too.

If you're interested in what go build is doing behind the scenes, you might like to use the following commands:

$ go list -deps . | sort -u     # List all packages that are used to build the executable
$ go build -a -x -o=/tmp/foo .  # Rebuild everything and show the commands that are run

Finally, if you run go build on a non-main package, it will be compiled in a temporary location and again, the result will be stored in the build cache. No executable is produced.

Cross-Compilation

This is one of my favorite features of Go.

By default go build will output a binary suitable for use on your current operating system and architecture. But it also supports cross-compilation, so you can generate a binary suitable for use on a different machine. This is particularly useful if you're developing on one operating system and deploying on another.

You can specify the operating system and architecture that you want to create the binary for by setting the GOOS and GOARCH environment variables respectively. For example:

$ GOOS=linux GOARCH=amd64 go build -o=/tmp/linux_amd64/foo .
$ GOOS=windows GOARCH=amd64 go build -o=/tmp/windows_amd64/foo.exe .

To see a list of all supported OS/architecture combinations you can run go tool dist list:

$ go tool dist list
aix/ppc64
android/386
android/amd64
android/arm
android/arm64
darwin/386
darwin/amd64
...

Hint: You can use Go's cross-compilation to create WebAssembly binaries.

For a bit more in-depth information about cross compilation I recommend reading this excellent post.

Using Compiler and Linker Flags

When building your executable you can use the -gcflags flag to change the behavior of the compiler and see more information about what it's doing. You can see a complete list of available compiler flags by running:

$ go tool compile -help

One flag that you might find interesting is -m, which triggers the printing of information about optimization decisions made during compilation. You can use it like this:

$ go build -gcflags="-m -m" -o=/tmp/foo . # Print information about optimization decisions

In the above example I used the -m flag twice to indicate that I want to print decision information two-levels deep. You can get simpler output by using just one.

Also, as of Go 1.10, compiler flags only apply to the specific packages passed to go build — which in the example above is the package in the current directory (represented by .). If you want to print optimization decisions for all packages including dependencies can use this command instead:

$ go build -gcflags="all=-m" -o=/tmp/foo .

As of Go 1.11, you should find it easier to debug optimized binaries than before. However, you can still use the flags -N to disable optimizations and -l to disable inlining if you need to. For example:

$ go build -gcflags="all=-N -l" -o=/tmp/foo .  # Disable optimizations and inlining

You can see a list of available linker flags by running:

$ go tool link -help

Probably the most well-known of these is the -X flag, which allows you to "burn in" a (string) value to a specific variable in your application. This is commonly used to add a version number or commit hash. For example:

$ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo .

For more information about the -X flag and some sample code see this StackOverflow question and this post and this post.

You may also be interested in using the -s and -w flags to strip debugging information from the binary. This typically shaves about 25% off the final size. For example:

$ go build -ldflags="-s -w" -o=/tmp/foo .  # Strip debug information from the binary

Note: If binary size is something that you need to optimize for you might want to use upx to compress it. See this post for more information.

Diagnosing Problems and Making Optimizations

Running and Comparing Benchmarks

A nice feature of Go is that it makes it easy to benchmark your code. If you're not familiar with the general process for writing benchmarks there are good guides here and here.

To run benchmarks you'll need to use the go test tool, with the -bench flag set to a regular expression that matches the benchmarks you want to execute. For example:

$ go test -bench=. ./...                        # Run all benchmarks and tests
$ go test -run=^$ -bench=. ./...                # Run all benchmarks (and no tests)
$ go test -run=^$ -bench=^BenchmarkFoo$ ./...   # Run only the BenchmarkFoo benchmark (and no tests)

I almost always run benchmarks using the -benchmem flag, which forces memory allocation statistics to be included in the output.

$  go test -bench=. -benchmem ./...

By default, each benchmark test will be run for a minimum of 1 second, once only. You can change this with the -benchtime and -count flags:

$ go test -bench=. -benchtime=5s ./...       # Run each benchmark test for at least 5 seconds
$ go test -bench=. -benchtime=500x ./...     # Run each benchmark test for exactly 500 iterations
$ go test -bench=. -count=3 ./...            # Repeat each benchmark test 3 times over

If the code that you're benchmarking uses concurrency, you can use the -cpu flag to see the performance impact of changing your GOMAXPROCS value (essentially, the number of OS threads that can execute your Go code simultaneously). For example, to run benchmarks with GOMAXPROCS set to 1, 4 and 8:

$ go test -bench=. -cpu=1,4,8 ./...

To compare changes between benchmarks you might want to use the benchcmp tool. This isn't part of the standard go command, so you'll need to install it like so:

$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/cmd/benchcmp

You can then use it like this:

$ go test -run=^$ -bench=. -benchmem ./... > /tmp/old.txt
# make changes
$ go test -run=^$ -bench=. -benchmem ./... > /tmp/new.txt
$ benchcmp /tmp/old.txt /tmp/new.txt
benchmark              old ns/op     new ns/op     delta
BenchmarkExample-8     21234         5510          -74.05%

benchmark              old allocs     new allocs     delta
BenchmarkExample-8     17             11             -35.29%

benchmark              old bytes     new bytes     delta
BenchmarkExample-8     8240          3808          -53.79%

Profiling and Tracing

Go makes it possible to create diagnostic profiles for CPU use, memory use, goroutine blocking and mutex contention. You can use these to dig a bit deeper and see exactly how your application is using (or waiting on) resources.

There are three ways to generate profiles:

If you have a web application you can import the net/http/pprof package. This will register some handlers with the http.DefaultServeMux which you can then use to generate and download profiles for your running application. This post provides a good explanation and some sample code.
For other types of applications, you can profile your running application using the pprof.StartCPUProfile() and pprof.WriteHeapProfile() functions. See the runtime/pprof documentation for sample code.
Or you can generate profiles while running benchmarks or tests by using the various -***profile flags like so:

$ go test -run=^$ -bench=^BenchmarkFoo$ -cpuprofile=/tmp/cpuprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -memprofile=/tmp/memprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -blockprofile=/tmp/blockprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -mutexprofile=/tmp/mutexprofile.out .

Note: Using the -***profile flags when running benchmarks or tests will result in a test binary being outputted to your current directory. If you want to output this to an alternative location you should use the -o flag like so:

$ go test -run=^$ -bench=^BenchmarkFoo$ -o=/tmp/foo.test -cpuprofile=/tmp/cpuprofile.out .

Whichever way you choose to create a profile, when profiling is enabled your Go program will stop about 100 times per second and take a snapshot at that moment in time. These samples are collected together to form a profile that you can analyze using the pprof tool.

My favorite way to inspect a profile is to use the go tool pprof -http command to open it in a web browser. For example:

$ go tool pprof -http=:5000 /tmp/cpuprofile.out

This will default to displaying a graph showing the execution tree for the sampled aspects of your application, which makes it possible to quickly get a feel for any resource usage 'hotspots'. In the graph above, we can see that the hotspots in terms of CPU usage are two system calls originating from ioutil.ReadFile().

You can also navigate to other views of the profile including top usage by function and source code.

If the amount of information is overwhelming, you might want to use the --nodefraction flag to ignore nodes that account for less than a certain percentage of samples. For example to ignore nodes that use appear in less than 10% of samples you can run pprof like so:

$ go tool pprof --nodefraction=0.1 -http=:5000 /tmp/cpuprofile.out

This makes the graph a lot less 'noisy' and if you zoom in on this screenshot, it's now much clearer to see and understand where the CPU usage hotspots are.

Profiling and optimizing resource usage is big, nuanced, topic and I've barely scratched the surface here. If you're interested in knowing more then I encourage you to read the following blog posts:

Another tool that you can use to help diagnose issues is the runtime execution tracer. This gives you a view of how Go is creating and scheduling goroutines to run, when the garbage collector is running, and information about blocking syscall/network/sync operations.

Again, you can generate trace from your tests or benchmarks, or use net/http/pprof to create and download a trace for your web application. You can then use go tool trace to view the output in your web browser like so:

$ go test -run=^$ -bench=^BenchmarkFoo$ -trace=/tmp/trace.out .
$ go tool trace /tmp/trace.out

Important: This is currently only viewable in Chrome/Chromium.

For more information about Go's execution tracer and how to interpret the output please see Rhys Hiltner's dotGo 2016 talk and this excellent blog post.

Checking for Race Conditions

I talked earlier about enabling Go's race detector during tests by using go test -race. But you can also enable it for running programs when building a executable, like so:

$ go build -race -o=/tmp/foo .

It's critical to note that race-detector-enabled binaries will use more CPU and memory than normal, so you shouldn't use the -race flag when building binaries for production under normal circumstances.

But you may want to deploy a race-detector-enabled binary on one server within a pool of many. Or use it to help track down a suspected race-condition by using a load-test tool to throw traffic concurrently at a race-detector-enabled binary.

By default, if any races are detected while the binary is running a log will be written to stderr. You can change this by using the GORACE environment variable if necessary. For example, to run the binary located at /tmp/foo and output any race logs to /tmp/race.<pid> you can use:

$ GORACE="log_path=/tmp/race" /tmp/foo

Managing Dependencies

You can use the go list tool to check whether a specific dependency has a newer version available like so:

$ go list -m -u github.com/alecthomas/chroma
github.com/alecthomas/chroma v0.6.2 [v0.6.3]

This will output the dependency name and version that you're currently using, followed by the latest version in square brackets [], if a newer one exists. You can also use go list to check for updates to all dependencies (and sub-dependencies) like so:

$ go list -m -u all

You can upgrade (or downgrade) a dependency to the latest version, specific tagged-release or commit hash with the go get command like so:

$ go get github.com/foo/bar@latest
$ go get github.com/foo/bar@v1.2.3
$ go get github.com/foo/bar@7e0369f

If the dependency you're updating has a go.mod file, then based on the information in this go.mod file, updates to any sub-dependencies will also be downloaded if necessary. If you use the go get -u flag, the contents of the go.mod file will be ignored and all sub-dependencies will be upgraded to their latest minor/patch version… even if the go.mod specifies a different version.

After upgrading or downgrading any dependencies it's a good idea to tidy your modfiles. And you might also want to run the tests for all packages to help check for incompatibilities. Like so:

$ go mod tidy
$ go test all

Occasionally you might want to use a local version of a dependency (for example, you need to use a local fork until a patch is merged upstream). To do this, you can use the go mod edit command to replace a dependency in your go.mod file with a local version. For example:

$ go mod edit -replace=github.com/alexedwards/argon2id=/home/alex/code/argon2id

This will add a replace rule to your go.mod file like so, and any future invocations of go run, go build etc will use the local version.

File: go.mod
module alexedwards.net/example

go 1.12

require github.com/alexedwards/argon2id v0.0.0-20190109181859-24206601af6c

replace github.com/alexedwards/argon2id => /home/alex/Projects/playground/argon2id

Once it's no longer necessary, you can remove the replace rule with the command:

$ go mod edit -dropreplace=github.com/alexedwards/argon2id

You can use the same general technique to import packages that exist only on your own file system. This can be useful if you're working on multiple modules in development at the same time, one of which depends on the other.

Note: If you don't want to use the go mod edit command, you can edit your go.mod file manually to make these changes. Either way will work.

Upgrading to a New Go Release

The go fix tool was originally released back in 2011 (when regular changes were still being made to Go's API) to help users automatically update their old code to be compatible with the latest version of Go. Since then, Go's compatibility promise means if you're upgrading from one Go 1.x version to a newer 1.x version everything should Just Work and using go fix should generally be unnecessary.

However, there are a handful of very specific issues that it does deal with. You can see a summary of them by running go tool fix -help. If you decide that you want or need to run go fix after upgrading, you should you run the following command, then inspect a diff of the changes before you commit them.

$ go fix ./...

Reporting Bugs

If you're confident that you've found an unreported issue with Go's standard library, tooling or documentation, you can use the go bug command to create a new Github issue.

$ go bug

This will open a browser window containing an issue pre-filled with your system information and reporting template.

Cheatsheet

Update 2019-04-19: @FedirFR has kindly made a cheatsheet based on this post. You can download it here or go-tooling-cheat-sheet.pdf.

If you enjoyed this blog post, don't forget to check out my new book about how to build professional web applications with Go!

Follow me on Twitter @ajmedwards.

All code snippets in this post are free to use under the MIT Licence.

bingoohuang commented 4 years ago

Carlo Alberto Ferraris提交了一个对math/rand库中的lockedSource优化的pr CL#191538,核心代码其实只有一行，却带来了相对的巨大的性能提升，让我们一起老看看这次的修改，学习一下代码的优化技巧，提高我们Go语言的底层优化经验。Carlo通过避免接口调用、允许内联、保持在同一个cacheline三种方式提升rngSource的性能。更多

bingoohuang commented 4 years ago

快速预览go的一篇文章，包括语法、安装、调度、并发等

bingoohuang commented 4 years ago

生产环境pprof，网络不直通（vpn 跳板机），telegraf cpu飙升到200%，咋办

# ps -ef|grep telegraf footsto+ 18552     1 99  2019 ?        101-20:55:59 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
# top -p 18552 -n 1
top - 15:34:11 up 560 days,  1:29,  1 user,  load average: 4.32, 4.32, 4.31
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 41.8 us, 23.6 sy,  0.0 ni, 32.7 id,  0.0 wa,  0.0 hi,  1.8 si,  0.0 st
KiB Mem :  8010196 total,   126864 free,  4718952 used,  3164380 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2962940 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
18552 footsto+  20   0  117628  59700   4888 S 200.0  0.7 146701:24 telegraf

146701:24 = 146701 minutes and 24 seconds ~= 146701/60/24 ~= 102 days, What does TIME+ in top mean?

Haven't looked at the source code of top, but it seems like 3019:57 in column TIME+ means 3019 minutes 57 seconds of accumulated CPU time.

    - 999:00.00 means 999 minutes
    - 1000:00 means a thousand minutes (no . separator at all)

I confirmed it on my system. I took an example process from top with 2529:38 then checked it with ps -fp PID which showed 1-18:09:38. The latter is one day and eighteen hours, which is approximately two and a half thousand minutes. Hence it equals the former.

下载安装go工具链linux版本, tar -C /usr/local -xzf go1.13.5.linux-amd64.tar.gz, export PATH=$PATH:/usr/local/go/bin

telegraf中开启 profiling, telegraf --config telegraf.conf --pprof-addr localhost:6060 Telegraf profiling

if *pprofAddr != "" {
    go func() {
        pprofHostPort := *pprofAddr
        parts := strings.Split(pprofHostPort, ":")
        if len(parts) == 2 && parts[0] == "" {
            pprofHostPort = fmt.Sprintf("localhost:%s", parts[1])
        }
        pprofHostPort = "http://" + pprofHostPort + "/debug/pprof"

        log.Printf("I! Starting pprof HTTP server at: %s", pprofHostPort)

        if err := http.ListenAndServe(*pprofAddr, nil); err != nil {
            log.Fatal("E! " + err.Error())
        }
    }()
}

go tool pprof开始采集数据

# go tool pprof http://localhost:6060/debug/pprof/profile
Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile
    Saved profile in /root/pprof/pprof.telegraf.samples.cpu.001.pb.gz
File: telegraf
Type: cpu
Time: Jan 7, 2020 at 4:31pm (CST)
Duration: 30.09s, Total samples = 1.74s ( 5.78%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 770ms, 44.25% of 1740ms total
Showing top 10 nodes out of 306
      flat  flat%   sum%        cum   cum%
     200ms 11.49% 11.49%      200ms 11.49%  runtime.futex
     190ms 10.92% 22.41%      230ms 13.22%  syscall.Syscall
      60ms  3.45% 25.86%       60ms  3.45%  runtime.epollwait
      60ms  3.45% 29.31%      140ms  8.05%  runtime.mapassign_faststr
      60ms  3.45% 32.76%      110ms  6.32%  runtime.scanobject
      50ms  2.87% 35.63%       50ms  2.87%  runtime.heapBitsSetType
      40ms  2.30% 37.93%       50ms  2.87%  runtime.findObject
      40ms  2.30% 40.23%       40ms  2.30%  runtime.mapiternext
      40ms  2.30% 42.53%       40ms  2.30%  runtime.memclrNoHeapPointers
      30ms  1.72% 44.25%      140ms  8.05%  github.com/bjca/telegraf/metric.(*metric).Fields
(pprof)

\# -http=:9100 就是再本地端⼝口9100的http服务上打开分析结果 
go tool pprof -http=:9100 http://127.0.0.1:6060/debug/pprof/profile

等待cpu高时，再观察。
参考资料
- Profiling Go Programs
  
  go tool pprof http://localhost:6060/debug/pprof/profile # 30-second CPU profile go tool pprof http://localhost:6060/debug/pprof/heap # heap profile go tool pprof http://localhost:6060/debug/pprof/block # goroutine blocking profile
- 使用google/gops,⼀般是在不允许开启http服务，或者应用在容器中
```
// 应⽤用程序中导⼊入gops代理理
import "github.com/google/gops/agent"

// 非侵⼊式加入代理代码(main.go)
if err := agent.Listen(agent.Options{
    ShutdownCleanup: true, // automatically closes on os.Interrupt
}); err != nil { 
    log.Fatal(err)
}

$ gops pprof-cpu (<pid>|<addr>) \# gops 命令使⽤用收集信息,对应的cpu profile分析命令
$ gops pprof-heap (<pid>|<addr>)
$ gops trace (<pid>|<addr>)
```
- Diagnostics
  
  Predefined profiles provided by the runtime/pprof package:
  
  cpu: CPU profile determines where a program spends its time while actively consuming CPU cycles (as opposed to while sleeping or waiting for I/O). heap: Heap profile reports memory allocation samples; used to monitor current and historical memory usage, and to check for memory leaks. threadcreate: Thread creation profile reports the sections of the program that lead the creation of new OS threads. goroutine: Goroutine profile reports the stack traces of all current goroutines. block: Block profile shows where goroutines block waiting on synchronization primitives (including timer channels). Block profile is not enabled by default; use runtime.SetBlockProfileRate to enable it. mutex: Mutex profile reports the lock contentions. When you think your CPU is not fully utilized due to a mutex contention, use this profile. Mutex profile is not enabled by default, see runtime.SetMutexProfileFraction to enable it.
- Getting started with Go CPU and memory profiling
```
package main

import (
    //...
    "github.com/pkg/profile"
)

func main() {
    // CPU profiling by default
    defer profile.Start().Stop()

    //...
}
```
  2017/08/03 14:26:28 profile: cpu profiling enabled, /var/...../cpu.pprof
  
  go tool pprof --pdf ~/go/bin/yourbinary /var/path/to/cpu.pprof > file.pdf go tool pprof --txt ~/go/bin/yourbinary /var/path/to/cpu.pprof > file.txt
```
package main

import (
    //...
    "github.com/pkg/profile"
)

func main() {
    // Memory profiling
    defer profile.Start(profile.MemProfile).Stop()

    //...
}
```
- [High Performance Go Workshop(https://dave.cheney.net/high-performance-go-workshop/sydney-2019.html)
- Continuous profiling in Go with Profefe, Profefe
  
  docker run -d -p 10100:10100 profefe/profefe
  
  You can push a profile in profefe:
```
$ curl -X POST \
    "http://localhost:10100/api/0/profiles?service=apid&type=cpu" \
    --data-binary @pprof.profefe.samples.cpu.001.pb.gz

    {"code":200,"body":{"id":"bo51acqs8snb9srq3p10","type":"cpu","service":"apid","created_at":"2019-12-30T15:18:11.361815452Z"}}
    You can retrieve it directly via its ID:

    $ go tool pprof http://localhost:10100/api/0/profiles/bo51acqs8snb9srq3p10

    Fetching profile over HTTP from http://localhost:10100/api/0/profiles/bo51acqs8snb9srq3p10
    Saved profile in /home/gianarb/pprof/pprof.profefe.samples.cpu.002.pb.gz
    File: profefe
    Type: cpu
    Time: Dec 23, 2019 at 4:06pm (CET)
    Duration: 30s, Total samples = 0
    There is a lot more you can do, when pushing a profile you can set key value pairs called labels and they can be used to query a portion of the profiles.
```
  You can use env=prod|test|dev or region=us|eu and so on.
  
  Retrieving a profile only via ID it’s not the unique way to visualize it. Profefe merges together profiles from the same type in a specific time range:
  
  GET /api/0/profiles/merge?service=<service>&type=<type>&from=<created_from>&to=<created_to>&labels=<key=value,key=value>
  
  It returns the raw compressed binary, it is compatible with go tool pprof as well as the single profile by id.

bingoohuang commented 4 years ago

GO重构

Apply transformations to Go code with eg

$ go get golang.org/x/tools/cmd/eg
$ eg -w -t T.template hello
=== /Users/jbd/src/hello/hello.go (1 matches)

Note: There are many .template files underneath the package for testing purposes but they can also be used as a reference how to write other transformation templates.

bingoohuang commented 4 years ago

围棋是一种策略性两人棋类游戏，中国古时称“弈”，西方名称“go”。

下围棋需要智慧。Playing go demands our wisdom.
基辛格将中国的外交政策比作是下围棋，一项需要耐心的围困博弈，胜利也只是相对的。 Kissinger likens Chinese diplomacy to the game of wei qi (equivalent to the Japanese game of go), a patient contest of encirclement in which victory is only relative.
中国围棋霸王赛 China Bawang Go Championship
中国围棋协会 Chinese Weiqi Association ; Chinese Go Association

bingoohuang commented 4 years ago

7种 Go 程序性能分析方法 - Dave Cheney

摘自1, 2

方法一：time
- shell 内置的 time
- GNU 实现的 time
- *BSD、macOS 的 time
- go tool 中的 -toolexec 参数
方法二：GODEBUG
插曲一：Profiler 是如何工作的？
- 性能分析注意事项
方法三：pprof
- CPU 性能分析
- 内存性能分析
- 阻塞性能分析
- 一次只分析一个东西
- 对函数分析
- 对整个应用分析
方法四：/debug/pprof
- 使用 pprof
  - 内存性能分析：
  - 阻塞性能分析
插曲二：Frame Pointer
方法五：perf
方法六：火焰图 (Flame Graph)
方法七：go tool trace

bingoohuang / blog

Golang的一些笔记 #84

Golang多版本安装

相关书籍

无缓冲读

一些库

JSON

JSON数据解析，直接给字段赋予json字符串，避免转义

JSON解析库

强制确保类型实现某个接口

在for循环里不要使用select + time.After的组合，有坑

以库为驱动开发的Golang工程结构，摘自Library driven development

稍微复杂一点的结构，摘自Go project structure to produce library and cli with the same name in single repository

看goroutine的数量

测试相关

参数化测试

测试集

BDD

Mock

Go Anti-Patterns : os.IsExist(err) != !os.IsNotExist(err) , [原文](https://stefanxo.com/go-anti-patterns-os-isexisterr-os-isnotexisterr/）

os.IsExist(err) is good for cases when you expect the file to not exist yet, but the file actually exists 👍

os.IsNotExist(err) is good for more common cases where you expect the file to exists, but it actually doesn't exist :

Further reading on this topic :

Channel 应用模式

Channel的特殊情况汇总

Lock/TryLock模式

or 信号模式

or channel by goroutine

or_done_channel模式

flat 模式

map/reduce 模式

Fan In 扇入模式

Fan Out 扇出模式

复制模式

分布模式

参考

Go调度器

GPM模型

Thanks

GOLANG私服

An Overview of Go's Tooling

Installing Tooling

Viewing Environment Information

Development

Running Code

Fetching Dependencies

Refactoring Code

Viewing Go Documentation

Testing

Running Tests

Profiling Test Coverage

Stress Testing

Testing all Dependencies

Pre-Commit Checks

Formatting Code

Performing Static Analysis

Linting Code

Tidying and Verifying your Dependencies

Build and Deployment

Building an Executable

Cross-Compilation

Using Compiler and Linker Flags

Diagnosing Problems and Making Optimizations

Running and Comparing Benchmarks

Profiling and Tracing

Checking for Race Conditions

Managing Dependencies

Upgrading to a New Go Release

Reporting Bugs

Cheatsheet

GO重构

7种 Go 程序性能分析方法 - Dave Cheney