Open bingoohuang opened 5 years ago
package main
import "fmt"
func main() {
// create unbuffered channel of int values with capacity of 1
ch := make(chan int)
select {
case ch <- 3:
// uncomment the following line to get this program work
// default:
}
fmt.Printf("ok\n")
}
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.main()
/tmp/056148531/main.go:10 +0x60
exit status 2
一些工具链
包管理,经常看到go.mod文件中,有一些indirect的包,是怎么被依赖进来的,可以用命令go mod why yourAwesomePackage
来查看。比如:
# bingoo @ 192 in ~/GitHub/loglineparser on git:master x [12:40:02]
$ go mod why gopkg.in/yaml.v2
# gopkg.in/yaml.v2
github.com/bingoohuang/loglineparser
github.com/araddon/dateparse
github.com/araddon/dateparse.test
github.com/simplereach/timeutils
gopkg.in/mgo.v2/bson
gopkg.in/mgo.v2/bson.test
gopkg.in/yaml.v2
As of the final 1.11 release, the go module cache (used for storing downloaded modules and source code), is in the $GOPATH/pkg/mod location (see the docs here). For clarification, the go build cache (used for storing recent compilation results) is in a different location.
This article, indicated that it's in the $GOPATH/src/mod, but in the timespan of the recent ~40 days, the golang team must have changed that target location. This message thread has some discussion on why the downloaded items ended up in $GOPATH/pkg.
You can also use the go mod download -json command to see the downloaded modules/source metadata and their location on your local disk. Example output below:
$ go mod download -json go: finding github.com/aws/aws-sdk-go v1.14.5 go: finding github.com/aws/aws-lambda-go v1.2.0
{
"Path": "github.com/aws/aws-lambda-go",
"Version": "v1.2.0",
"Info": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.info",
"GoMod": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.mod",
"Zip": "/go/pkg/mod/cache/download/github.com/aws/aws-lambda-go/@v/v1.2.0.zip",
"Dir": "/go/pkg/mod/github.com/aws/aws-lambda-go@v1.2.0",
"Sum": "h1:2f0pbAKMNNhvOkjI9BCrwoeIiduSTlYpD0iKEN1neuQ=",
"GoModSum": "h1:zUsUQhAUjYzR8AuduJPCfhBuKWUaDbQiPOG+ouzmE1A="
}
{
"Path": "github.com/aws/aws-sdk-go",
"Version": "v1.14.5",
"Info": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.info",
"GoMod": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.mod",
"Zip": "/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.14.5.zip",
"Dir": "/go/pkg/mod/github.com/aws/aws-sdk-go@v1.14.5",
"Sum": "h1:+l1m6QH6LypE2kL0p/G0Oh7ceCv+IVQ1h5UEBt2xjjU=",
"GoModSum": "h1:ZRmQr0FajVIyZ4ZzBYKG5P3ZqPz9IHG41ZoMu1ADI3k="
}
package main
import (
"encoding/json"
"fmt"
)
func main() {
s := `[` + `{"name":"bingoo"},{"name":"dingoo"}` + `]`
var arr []interface{}
if err := json.Unmarshal([]byte(s), &arr); err != nil {
panic(err)
}
m := map[string]interface{}{"key1": arr, "key2": s, "key3": json.RawMessage([]byte(s))}
jso, err := json.Marshal(m)
if err != nil {
panic(err)
}
// {"key1":[{"name":"bingoo"},{"name":"dingoo"}],"key2":"[{\"name\":\"bingoo\"},{\"name\":\"dingoo\"}]","key3":[{"name":"bingoo"},{"name":"dingoo"}]}
fmt.Println(string(jso))
}
Popular replacements for standard library packages:
import "github.com/thedevsaddam/gojsonq"
func main() {
const json = `{"name":{"first":"Tom","last":"Hanks"},"age":61}`
name := gojsonq.New().FromString(json).Find("name.first")
println(name.(string)) // Tom
}
Go语言中,类型实现某个接口 ,只要实现了该接口中所有定义的方法即可,没有像Java中的implements关键字,是一种契约式精神的体现,或者说鸭子类型。那有没有强制某个类型必须某个接口的写法呢,刚刚在翻阅jons.RawMessage中看到了以下两行代码,刚开始感觉迷惑,仔细思考后,便了解了其意图:
var _ Marshaler = (*RawMessage)(nil)
var _ Unmarshaler = (*RawMessage)(nil)
带上下文的代码如下:
// RawMessage is a raw encoded JSON value.
// It implements Marshaler and Unmarshaler and can
// be used to delay JSON decoding or precompute a JSON encoding.
type RawMessage []byte
// MarshalJSON returns m as the JSON encoding of m.
func (m RawMessage) MarshalJSON() ([]byte, error) {
if m == nil {
return []byte("null"), nil
}
return m, nil
}
// UnmarshalJSON sets *m to a copy of data.
func (m *RawMessage) UnmarshalJSON(data []byte) error {
if m == nil {
return errors.New("json.RawMessage: UnmarshalJSON on nil pointer")
}
*m = append((*m)[0:0], data...)
return nil
}
var _ Marshaler = (*RawMessage)(nil)
var _ Unmarshaler = (*RawMessage)(nil)
本来想给朋友宣称一下,西游可能有Bug,我都已经被挤下线了,但是还能继续谷歌(假百度),然后随便输入了一个golang oom作为搜索条件:
结果就把这个搜索结果页面扔到一边,去干其他事情去了。
等回过头来,打算关闭这个标签页的时候,发现结果第一条,还是有点兴趣的,就点进去看了,最后有这么一句话:
有经验的gopher都知道,在for循环里不要使用select + time.After的组合,有坑。 当使用golang过程中,遇到性能和内存gc问题,都可以使用golang tool pprof来排查分析问题。
然后,想到自己也经常这么用(for select time.After),很可能踩坑。然后去继续探索了一下,为什么可能会有坑 ,然后将代码中的for select time.After全部重构成 time.Timer + for来预防坑。
Don't use time.Tick to prevent leaks
就是因为在循环中,不停的创建新的计时器,而每个计时器都会开启内部协程。再看看计时器函数的官方注释:
// Tick is a convenience wrapper for NewTicker providing access to the ticking
// channel only. While Tick is useful for clients that have no need to shut down
// the Ticker, be aware that without a way to shut it down the underlying
// Ticker cannot be recovered by the garbage collector; it "leaks".
// Unlike NewTicker, Tick will return nil if d <= 0.
func Tick(d Duration) <-chan Time {
if d <= 0 {
return nil
}
return NewTicker(d).C
}
一些代码检查工具
golangci-lint run
, 来自这里,安装GO111MODULE=on go get github.com/golangci/golangci-lint/cmd/golangci-lint@v1.16.0
golangci-lint run --exclude="cyclomatic complexity" --exclude-use-default=false --enable=golint --enable=gocyclo --enable=goconst --enable=unconvert ./...
golangci-lint run --enable-all
GO111MODULE=off go get honnef.co/go/tools/cmd/...
,运行staticcheck ./...
检查依赖是否有升级
go get -u github.com/psampaz/go-mod-outdated
go list -u -m -json all | go-mod-outdated -direct
example output
+---------------------------------+----------------------------------------------------------------+------------------------------------+--------+------------------+
| MODULE | VERSION | NEW VERSION | DIRECT | VALID TIMESTAMPS |
+---------------------------------+----------------------------------------------------------------+------------------------------------+--------+------------------+
| github.com/alecthomas/template | v0.0.0-20160405071501-a0175ee3bccc | | true | true |
| github.com/bingoohuang/gou | v0.0.0-20190604082926-bf3d9b2b55aa5840c442284656ed6b15aedc5a25 | v0.0.0-20190604082926-be6100942b5a | true | false |
| github.com/bingoohuang/now | v0.0.0-20190604021600-70970d3ad0e7 | | true | true |
| github.com/bingoohuang/statiq | v0.2.1 | | true | true |
| github.com/dgraph-io/badger | v2.0.0-rc.2+incompatible | v1.5.4 | true | false |
| github.com/gchaincl/dotsql | v0.1.0 | | true | true |
| github.com/gin-contrib/sessions | v0.0.0-20190226023029-1532893d996f | v0.0.0-20190512062852-3cb4c4f2d615 | true | true |
| github.com/gin-gonic/gin | v1.4.0 | | true | true |
| github.com/go-sql-driver/mysql | v1.4.1 | | true | true |
| github.com/lib/pq | v1.0.0 | v1.1.1 | true | true |
| github.com/mattn/go-sqlite3 | v1.10.0 | | true | true |
| github.com/patrickmn/go-cache | v2.1.0+incompatible | | true | true |
| github.com/pkg/errors | v0.8.1 | | true | true |
| github.com/sirupsen/logrus | v1.4.2 | | true | true |
| github.com/spf13/pflag | v1.0.3 | | true | true |
| github.com/spf13/viper | v1.3.2 | v1.4.0 | true | true |
| github.com/stretchr/testify | v1.3.0 | | true | true |
| github.com/swaggo/swag | v1.4.1 | v1.5.0 | true | true |
| github.com/thoas/go-funk | v0.4.0 | | true | true |
+---------------------------------+----------------------------------------------------------------+------------------------------------+--------+------------------+
Go linux安装
下载,解压缩
[vagrant@bogon ~]$ curl -O https://dl.google.com/go/go1.12.5.linux-amd64.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 122M 100 122M 0 0 349k 0 0:05:57 0:05:57 --:--:-- 356k
[vagrant@bogon ~]$ tar -xzf go1.12.5.linux-amd64.tar.gz
[vagrant@bogon ~]$ ls
base.sh cleanup.sh go go1.12.5.linux-amd64.tar.gz puppet.sh vagrant.sh virtualbox.sh zerodisk.sh
设置环境变量PATH
[vagrant@bogon ~]$ echo "export PATH=\$PATH:/home/vagrant/go/bin" >> ~/.bashrc; source ~/.bashrc
[vagrant@bogon ~]$ go version
go version go1.12.5 linux/amd64
首先是执行go get golang.org/dl/gotip
安装tip的辅助命令,然后通过执行gotip download
下载真正的tip版本工具。下载完成之后,就可以通过totip命令来编译和运行Go程序了。
➜ kerb git:(master) ✗ go get golang.org/dl/gotip
go: finding golang.org/dl/gotip latest
go: finding golang.org/dl latest
go: downloading golang.org/dl v0.0.0-20190507014322-219d744c5398
go: extracting golang.org/dl v0.0.0-20190507014322-219d744c5398
➜ kerb git:(master) ✗ gotip download
Cloning into '/Users/bingoobjca/sdk/gotip'...
remote: Counting objects: 9475, done
remote: Finding sources: 100% (9475/9475)
remote: Total 9475 (delta 1010), reused 5862 (delta 1010)
package main
Receiving objects: 100% (9475/9475), 22.11 MiB | 363.00 KiB/s, done.
Resolving deltas: 100% (1010/1010), done.
Checking out files: 100% (8598/8598), done.
HEAD is now at f2a4c13 errors: clarify doc for As
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
---
Installed Go for darwin/amd64 in /Users/bingoobjca/sdk/gotip
Installed commands in /Users/bingoobjca/sdk/gotip/bin
Success. You may now run 'gotip'!
➜ kerb git:(master) ✗ gotip version
go version devel +f2a4c13 Tue Jun 11 21:50:05 2019 +0000 darwin/amd64
GOPROXY
# Enable the go modules feature
export GO111MODULE=on
# Set the GOPROXY environment variable
export GOPROXY=https://goproxy.io
export GO111MODULE=on
export GOPROXY=https://proxy.golang.org
GOPROXY=direct,https://127.0.0.1:12333,https://goproxy.cn,https://goproxy.io,https://mirrors.aliyun.com/goproxy,https://athens.azurefd.net
go env -w GOSUMDB="off"
Go 语言诞生时,我们称它为系统编程语言,我有点遗憾,因为很多人因此认为它是一种操作系统编写语言。我们应该称它为服务编写语言,这是我们真正想做的。现在我想明白了,Go 是云基础架构语言,因为系统编程的另一个定义是云中运行的东西。
When we first announced Go, we called it a systems programming language, and I slightly regret that because a lot of people assumed it was an operating systems writing language. What we should have called it is a server writing language, which is what we really thought of it as. Now I understand that what we have is a cloud infrastructure language. Another definition of systems programming is the stuff that runs in the cloud.
-- Rob Pike
今天我探索了一下golang的两个方面:
然后我发现了函数选项(Functional Options)模式, GIST上有一个最小的例子。
函数选项模式是由Rob Pike提出,并由Dave Cheney等推广开,它优雅地解决了go语言中默认参数问题。
下面总结一下,函数选项模式有哪些优点:
推而广之:类似结构体中变量的赋值都可以效仿之。
溜开源项目
创建的对象,里面已经启动了一个后台的死循环的go协程,当对象不再被使用时,因为背后的go协程还一直在跑,导致对象不能被gc回收,咋办?,参见go-cache的实现
func New(defaultExpiration, cleanupInterval time.Duration) *Cache {
items := make(map[string]Item)
return newCacheWithJanitor(defaultExpiration, cleanupInterval, items)
}
func newCacheWithJanitor(de time.Duration, ci time.Duration, m map[string]Item) *Cache {
c := newCache(de, m)
// This trick ensures that the janitor goroutine (which--granted it
// was enabled--is running DeleteExpired on c forever) does not keep
// the returned C object from being garbage collected. When it is
// garbage collected, the finalizer stops the janitor goroutine, after
// which c can be collected.
C := &Cache{c}
if ci > 0 {
runJanitor(c, ci)
runtime.SetFinalizer(C, stopJanitor)
}
return C
}
func runJanitor(c *cache, ci time.Duration) {
j := &janitor{
Interval: ci,
stop: make(chan bool),
}
c.janitor = j
go j.Run(c)
}
Moving the main.go file out of your root allows you to build your application from the perspective of a library. Your application binary is simply a client of your application’s library. Sometimes you might want users to interact in multiple ways so you create multiple binaries. For example, if you had an “adder” package that that let users add numbers together, you may want to release a command line version as well as a web version. You can easily do this by organizing your project like this:
adder/
adder.go
cmd/
adder/
main.go
adder-server/
main.go
Users can install your “adder” application binaries with “go get” using an ellipsis:
$ go get github.com/benbjohnson/adder/...
And voila, your user has “adder” and “adder-server” installed!
Of course if your project is more complex, you may create further packages under the project root, and it may have multiple commands (multiple main packages), e.g.:
host.com/project/
cmd/
project/
project.go
prjtool/
prjtool.go
packagex/
x.go
packagey/
y.go
filea.go
fileb.go
An example following this layout is the very popular Go Delve debugger (4.5k stars currently).
This post discusses an end-to-end example that covers:
The full source code for this example is available on Github. cgo-callback.zip
宽进严出原则:
Be conservative in what you send, be liberal in what you accept
- Robustness Principle
在golang上的应用:
Return concrete types, receive interfaces as parameters
选哪个?
func New() *os.File
func New() io.ReadWriteCloser
func New() io.Writer
func New() interface{}
根据宽进严出
: 选 func New() *os.File
➜ golang-trial git:(master) ✗ GODEBUG=schedtrace=1000 gohttpd
SCHED 0ms: gomaxprocs=12 idleprocs=10 threads=4 spinningthreads=1 idlethreads=0 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 1009ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 2013ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 3019ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 4026ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 5033ms: gomaxprocs=12 idleprocs=12 threads=7 spinningthreads=0 idlethreads=3 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 6036ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=21 runqueue=152 [4 1 11 0 2 1 3 10 4 5 1 9]
SCHED 7042ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=209 [12 7 15 2 18 15 0 13 19 3 2 16]
SCHED 8046ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=170 [0 12 10 8 4 5 0 3 4 5 1 5]
SCHED 9055ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=21 runqueue=234 [3 0 15 9 3 11 1 10 5 6 10 3]
SCHED 10056ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=221 [1 13 15 9 5 2 0 0 8 12 4 10]
SCHED 11058ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=18 runqueue=159 [9 12 4 13 4 9 0 10 11 1 1 12]
SCHED 12061ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=19 runqueue=204 [0 16 3 18 7 16 3 2 17 12 18 13]
SCHED 13062ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=1 idlethreads=23 runqueue=237 [16 12 3 13 7 6 0 1 1 15 9 21]
SCHED 14062ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=17 runqueue=201 [16 1 1 11 14 2 5 10 2 0 0 16]
SCHED 15072ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=22 runqueue=26 [0 0 0 0 0 0 0 0 0 0 0 0]
SCHED 16080ms: gomaxprocs=12 idleprocs=0 threads=38 spinningthreads=0 idlethreads=21 runqueue=202 [10 16 18 0 12 0 19 13 9 17 0 1]
SCHED 17087ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=15 runqueue=167 [8 3 12 14 14 14 4 17 1 6 15 6]
SCHED 18092ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=209 [18 4 0 17 4 9 0 10 0 3 5 16]
SCHED 19093ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=21 runqueue=198 [16 17 3 6 4 1 14 6 18 18 10 21]
SCHED 20093ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=24 runqueue=232 [7 12 13 5 3 7 5 2 8 1 9 8]
SCHED 21100ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=24 runqueue=206 [16 1 14 9 16 7 16 13 5 1 8 4]
SCHED 22110ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=23 runqueue=249 [8 6 8 3 10 9 12 8 6 11 0 3]
SCHED 23113ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=20 runqueue=223 [0 5 17 2 5 3 0 0 0 14 9 2]
SCHED 24116ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=1 idlethreads=22 runqueue=219 [1 0 9 3 1 2 13 10 1 1 18 12]
SCHED 25124ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=23 runqueue=199 [14 12 15 6 2 1 5 0 11 1 15 0]
SCHED 26126ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=21 runqueue=169 [1 12 6 11 8 5 8 5 5 10 6 4]
SCHED 27126ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=20 runqueue=232 [4 5 9 0 1 1 1 1 5 1 11 4]
SCHED 28129ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=21 runqueue=243 [9 11 7 3 7 14 3 12 2 12 12 1]
SCHED 29136ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=20 runqueue=58 [0 4 0 1 0 0 2 0 0 3 0 0]
SCHED 30144ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=153 [0 9 7 1 4 2 10 2 5 0 0 1]
SCHED 31151ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=216 [10 4 9 5 14 7 1 10 7 18 8 5]
SCHED 32159ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=24 runqueue=203 [5 8 2 0 5 1 1 14 3 0 14 12]
SCHED 33164ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=22 runqueue=212 [14 13 5 19 0 13 1 8 15 9 11 2]
SCHED 34174ms: gomaxprocs=12 idleprocs=0 threads=40 spinningthreads=0 idlethreads=23 runqueue=266 [6 4 12 12 1 0 8 4 0 3 0 3]
参考:
在写测试时,有时需要在测试之前或之后进行额外的设置(setup)或拆卸(teardown);有时,测试还需要控制在主线程上运行的代码。为了支持这些需求,testing 提供了 TestMain 函数:
func TestMain(m *testing.M)
package mytestmain
import (
"flag"
"fmt"
"os"
"testing"
)
var db struct {
Dns string
}
func TestMain(m *testing.M) {
db.Dns = os.Getenv("DATABASE_DNS")
if db.Dns == "" {
db.Dns = "root:123456@tcp(localhost:3306)/?charset=utf8&parseTime=True&loc=Local"
}
flag.Parse()
exitCode := m.Run()
db.Dns = ""
// 退出
os.Exit(exitCode)
}
func TestDatabase(t *testing.T) {
fmt.Println(db.Dns)
}
func TestAdd(t *testing.T) {
tests := []struct{
name string
first int64
second int64
expected int64
} {
{
name: "HappyPath":
first: 2,
second: 3,
expected: 5,
},
{
name: "NegativeNumber":
first: -1,
second: -1,
expected: -2,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
assert.Equal(t, test.expected, Add(test.first, test.second))
})
}
}
import (
"testing"
"github.com/stretchr/testify/suite"
)
type ExampleTestSuite struct {
suite.Suite
VariableThatShouldStartAtFive int
}
func (suite *ExampleTestSuite) SetupTest() {
suite.VariableThatShouldStartAtFive = 5
}
func (suite *ExampleTestSuite) TestExample() {
suite.Equal(suite.VariableThatShouldStartAtFive, 5)
}
func TestExampleTestSuite(t *testing.T) {
suite.Run(t, new(ExampleTestSuite))
}
var _ = Describe("Book", func() {
var (
book Book
err error
)
BeforeEach(func() {
book, err = NewBookFromJSON(`{
"title":"Les Miserables",
"author":"Victor Hugo",
"pages":1488
}`)
})
Describe("loading from JSON", func() {
Context("when the JSON fails to parse", func() {
BeforeEach(func() {
book, err = NewBookFromJSON(`{
"title":"Les Miserables",
"author":"Victor Hugo",
"pages":1488oops
}`)
})
It("should return the zero-value for the book", func() {
Expect(book).To(BeZero())
})
It("should error", func() {
Expect(err).To(HaveOccurred())
})
})
})
})
猴子补丁其实就是一个大杀器了,bouk/monkey 能够通过替换函数指针的方式修改任意函数的实现,所以如果上述的几种方法都不能满足我们的需求,我们就只能够通过猴子补丁这种比较 hack 的方法 Mock 依赖了,:
func main() {
monkey.Patch(fmt.Println, func(a ...interface{}) (n int, err error) {
s := make([]interface{}, len(a))
for i, v := range a {
s[i] = strings.Replace(fmt.Sprint(v), "hell", "*bleep*", -1)
}
return fmt.Fprintln(os.Stdout, s...)
})
fmt.Println("what the hell?") // what the *bleep*?
}
不要在单元测试之外的地方使用猴子补丁,我们应该只在必要的时候使用这种方法,例如依赖的第三方库没有提供 interface 或者修改 time.Now 以及 rand.Int63n 等内置函数的返回值用于测试时。
从理论上来说,通过猴子补丁这种方式我们能够在运行时 Mock Go 语言中的一切函数,这也为我们提供了单元测试 Mock 依赖的最终解决方案。
go module upgrades dependencies:
go get -u
(without any arguments) now only upgrades the direct and indirect dependencies of your current package, and no longer examines your entire module.go get -u ./...
from your module root upgrades all the direct and indirect dependencies of your module, and now excludes test dependencies.go get -u -t ./...
is similar, but also upgrades test dependencies.GO语言的历史,非常好的演讲,值得一看。
编程语言发展的四波浪潮:
Go恢复了早期语言的简单性和灵活性,增加了现代语言的安全性和开发友好性。Go以一种非常真实的方式复兴了许多伟大的想法,这些想法终于准备就绪。
Go给人的感觉就像是来自60年代,70年代,80年代,90年代,00s,10年代的语言……Steve Francia 2019
Go的设计哲学
如有疑问,请将其排除在外。- Joshua Bloch:关于设计的对话– 2002
当我们三个人开始时,这纯粹是研究。…我们从一个想法开始,即我们三个人都必须针对该语言的每个特性进行讨论,因此,无论出于何种原因,都不会在该语言中放入多余的垃圾。- 肯·汤普森(Ken Thompson)访谈– 2011年,肯从Bell Labs学习了这种做法 有两种构建软件设计的方法。一种方法是使其变得如此简单,以至于显然没有缺陷。另一种方法是使其变得如此复杂,以至于没有明显的缺陷。- 托尼·霍尔(Tony Hoare)皇帝的旧衣服-1981年,Go采取了第一种方法,而大多数其他语言都采用第二种方法。
当您处于语言的设计阶段时,您将需要进行频繁且有时是巨大的更改。朝着这个期望前进,并围绕它建立您的流程。
CentOS 上安装golang
curl -LO https://studygolang.com/dl/golang/go1.13.4.linux-amd64.tar.gz
sudo tar -C /usr/local/ -xzvf go1.13.4.linux-amd64.tar.gz
/usr/local/go/bin/go version
vi .bash_profile
, export PATH=$PATH:/usr/local/go/bin
export http_proxy=http://192.168.33.1:9999; export https_proxy=http://192.168.33.1:9999;
go get -u -v golang.org/x/lint/golint
go get -u -v golang.org/x/lint/golint
curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s -- -b $(go env GOPATH)/bin v1.21.0
如何有效地缩小go可执行文件大小
总结:use go build -ldflags="-s -w" ./...
and then use UPX with --best --lzma binfile
.
go build | go build -ldflags="-s -w" | upx hello | upx --brute hell |
---|---|---|---|
2.0M | 1.6M (20% off) | 1.6M(62% off,70%off total) | 484K(21% off, 76% off total) |
// hello.go
package main
import "fmt"
func main() {
fmt.Printf("hello, world\n")
}
$ go version
go version go1.13.4 darwin/amd64
$ go build hello.go
$ ls -lh
-rwxr-xr-x 1 bingoo staff 2.0M 11 12 22:51 hello
-rw-r--r-- 1 bingoo staff 86B 11 12 22:50 hello.go
$ go build -ldflags="-s -w" hello.go
$ ls -lh
-rwxr-xr-x 1 bingoo staff 1.6M 11 12 22:51 hello
$ upx hello
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2018
UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018
File size Ratio Format Name
-------------------- ------ ----------- -----------
1665904 -> 630800 37.87% macho/amd64 hello
Packed 1 file.
$ ls -lh
-rwxr-xr-x 1 bingoo staff 616K 11 12 22:51 hello
$ go build -ldflags="-s -w" hello.go
$ upx --brute hello
Ultimate Packer for eXecutables
Copyright (C) 1996 - 2018
UPX 3.95 Markus Oberhumer, Laszlo Molnar & John Reiser Aug 26th 2018
File size Ratio Format Name
-------------------- ------ ----------- -----------
1665904 -> 495632 29.75% macho/amd64 hello
Packed 1 file.
$ ls -lh
-rwxr-xr-x 1 bingoo staff 484K 11 12 22:52 hello
Golang executables embed lots of symbolic information:
To reproduce my results, you can do the following:
go get -u -t -v github.com/taskcluster/taskcluster-lib-artifact-go
cd $GOPATH/src/github.com/taskcluster/taskcluster-lib-artifact-go
git checkout 6f133d8eb9ebc02cececa2af3d664c71a974e833
time (go build) && wc -c ./artifact
time (go build && strip ./artifact) && wc -c ./artifact
time (go build -ldflags="-s") && wc -c ./artifact
time (go build -ldflags="-w") && wc -c ./artifact
time (go build -ldflags="-s -w") && wc -c ./artifact
time (go build && upx -1 ./artifact) && wc -c ./artifact
time (go build && upx -9 ./artifact) && wc -c ./artifact
time (go build && strip ./artifact && upx -1 ./artifact) && wc -c ./artifact
time (go build && strip ./artifact && upx --brute ./artifact) && wc -c ./artifact
time (go build && strip ./artifact && upx --ultra-brute ./artifact) && wc -c ./artifact
time (go build && strip && upx -9 ./artifact) && wc -c ./artifact
man strip
: strip removes or modifies the symbol table attached to the output of the assembler and link editor. This is useful to save space after a program has been debugged and to limit dynamically bound symbols.strip用于脱掉文件的衣服, 文件会变小, 其中的符号信息会失去。 那这个strip有什么用呢? 很有用的! 原来的a.out比较大, 可以执行。 在strip之后, 文件变小了, 仍然可以执行, 这就就节省了很多空间。 strip不仅仅可以针对可执行文件, 还能针对目标文件和动态库等。 在实际的开发中, 经常需要对动态库.so进行strip操作, 减少占地空间。 而在调试的时候(比如用addr2line), 就需要符号了。 因此, 通常的做法是: strip前的库用来调试, strip后的库用来实际发布, 他们两者有对应关系。 一旦发布的strip后的库出了问题, 就可以找对应的未strip的库来定位。
出现问题,依赖高版本的glibc包,目标机器centos6.9,没有,报错。
$ ./rigaga
./rigaga: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./rigaga)
$ ll /lib64/libc.so.6
lrwxrwxrwx. 1 root root 12 6月 8 2018 /lib64/libc.so.6 -> libc-2.12.so
$ go build -o sysinfo cmd/sysinfo/main.go
$ ldd sysinfo
linux-vdso.so.1 => (0x00007ffdff179000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b1cff7fc000)
libc.so.6 => /lib64/libc.so.6 (0x00002b1cffa18000)
/lib64/ld-linux-x86-64.so.2 (0x00002b1cff5d8000)
[footstone@fs01-192-168-126-182 sysinfo]$ go build -o sysinfo -ldflags '-linkmode "external" -extldflags "-static"' cmd/sysinfo/main.go
# command-line-arguments
/tmp/go-link-033563708/000002.o: In function `mygetgrouplist':
/workdir/go/src/os/user/getgrouplist_unix.go:16: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetgrgid_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetgrnam_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetpwnam_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000001.o: In function `mygetpwuid_r':
/workdir/go/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-033563708/000008.o: In function `_cgo_7e1b3c2abc8d_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:57: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
$ ldd sysinfo
not a dynamic executable
Site:
Before doing go build Set CGO_LDFLAGS
Dynamic: export CGO_LDFLAGS="-Xlinker -rpath=/path/to/another_glibc/lib"
Static: export CGO_LDFLAGS="-Xlinker -rpath=/path/to/another_glibc/lib -static"
CGO_LDFLAGS lets you set GCC-like ld flags for Go.
如果汽车像电脑一样发展的话,会是什么样子?If cars developed at the same pace as computers, they’d be this fast
Example 1: We want to create a symlink, but the target path exists already :
if _, err := os.Symlink("/old/path", "/path/to/whatever"); os.IsExist(err) {
// error happened, can't Symlink
// can't create symlink because /path/to/whatever already exists
}
Example 2: We want to stat a file, but it doesn't exist :
if _, err := os.Stat("/path/to/whatever"); os.IsNotExist(err) {
// error happened, can't Stat!
// /path/to/whatever does not exist
}
Anti-Pattern
: And here comes the big BUT :
// Anti-pattern : We want to stat a file, and continue if it exists :
if _, err := os.Stat("/file/that/exists"); os.IsExist(err) {
// will never trigger!
// why? because os.Stat runs normally if file exists.
// it's expected behaviour for os.Stat, so it doesn't throw an error
// os.IsExist() does not receive an error ( it's nil ), so it can't tell you
// if the error message was "file not found"
}
Instead we want to stat a file, and continue if it exists :
_, err := os.Stat("/file/that/exists");
if err != nil {
if os.IsNotExist(err) {
// file does not exist, do something
} else {
// more serious errors
}
}
// file exists.. continue with code here, or in else statement or specify if err == nil { // do something }
os.Symlink("/path/that/exists", "/path/to/symlink/target")
// os.IsExist(err) will trigger when target exists already
os.Mkdir(target)
// os.IsExist(err) will trigger because target path already exists
os.OpenFile(target, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0600)
// os.IsExist(err) will trigger when target exists because O_EXCL means that file should not exist yet
os.Chdir()
os.Stat()
os.Open()
os.OpenFile() // without os.O_EXCL
os.Chmod()
os.Chown()
os.Close()
os.Read()
os.ReadAt()
os.ReadDir()
os.Readdirnames()
os.Seek()
os.Truncate()
os.Write()
os.WriteAt()
os.WriteString()
虽然go支持跨平台编译,例如在mac系统编译linux版本,大部分情况下是没有问题的,但是不是所有情况,例如,引用sqlite库时,就存在问题,就必须在linux环境下去编译了。linux环境,本机可以使用vagrant加载一个linux,也可以使用docker镜像来。下面的例子来自reddit docker_go_deployment
create Dockerfile:
# build stage
FROM golang:1.13.4-alpine AS build
# env
RUN mkdir -p /app
WORKDIR /app
# tools
RUN apk add --no-cache git
# build src
COPY go.mod .
COPY go.sum .
RUN go mod download
# app src
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /bin/app
# result stage
FROM scratch
COPY --from=build /bin/app /bin/app
ENTRYPOINT ["/bin/app" "--port" "8080"]
for testing you can use docker-compose (maybe you need to install it first)
create docker-compose.yml
version: '3'
services:
your_service_name:
build: .
ports:
- "8080:8080"
restart: always
volumes:
- .:/app
initialize your project
go mod init <your-package-name>
get all needed modules into your vendor folder
go mod vendor
build your service(s)
docker-compose build
test your services locally, check localhost:8080
docker-compose up
~ | ni | not empty | empty | full | not full |
---|---|---|---|---|---|
Receive | block | value | block | value | value |
Send | block | write value | write value | block | write value |
close | panic | closed, drained read, return zero value | closed, return zero value | closed, drained read, return zero value | closed, drained read, return zero value |
从多个channel读取一个信号, 一旦读取到一个信号,则不再读取。
比如向多个服务器发送相同的http request,每个请求的结果放在单独的一个channel中, 只要其中一个服务器返回结果,则其它请求就被忽略。
最简单的方式就是为每个channel启动一个goroutine, 每个goroutine读取自己负责的channel,一旦读取到一个信号,就关闭返回的channel。 显然,为每个channel启动一个goroutine太浪费了,虽然goroutine是一种轻量级的实现,但是如果数量巨大的情况下也会导致资源的大量占用以及调度上的性能低下。
or_channel_go 基于递归的方式实现, 使用依次递归的方式
or_channel_rec 基于递归的方式实现, 使用分而治之的方式
or_channel_reclect 基于反射的方式
与上面的or 信号模式不同, or done channel模式是从一个channel中读取数据,只有当channel被关闭,或者done 信号channel被关闭的时候,读取操作才退出。
如果将done这个信号channel换成 context,则可以依靠 context.WithCancel 来cancel读取,和这个模式类似。
flat 将多个channels平展成一个channels。 与Fan In不同的是,输入的channels是从一个channel中读取出来的,而Fan In模式中的channels是一个channel slice。
mapreduce
将多个channel合并成一个channel
fanIn
将一个Channel分成多个Channel。 有两种情况, 一种是每个channel都包含同样的数据(复制模式), 另一种将原数据均匀分布到各输出channel中(分布模式)
+-------+ +-------+
| KSE | | KSE |
+-------+ +-------+
| | 内核空间
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| | 用户空间
+-------+ +-------+
| M | | M |
+-------+ +-------+
| | | |
+------+ +------+ +------+ +------+
| P | | P | | P | | P |
+------+ +------+ +------+ +------+
| | | | | | | | |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
| G | | G | | G | | G | | G | | G | | G | | G | | G |
+---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+
How to point Go module dependency in go.mod to a latest commit in a repo?
Just 'go get' at the commit hash you want:
go get github.com/someone/some_module@af044c0995fe
'go get' will correctly update the dependency files (go.mod, go.sum).
Erlang的作者Armstrong在其博士论文中证明,单个机器自身,由于对其硬件出错的不可挽回,本质上是不可靠的。但独立机器构成的网络,可以互相监督和备份,在容错系统软件的支持下,完全有能力大大提高整体的可靠性到九个9,也就是理论上几乎完全可靠。
由于有了go func,除了在main大厅外,Go还可以轻易的安排非常多的小房间,和main大厅里的同时func。这种互不打扰的群体func行为史称并发。 runtime就是龟婆,提供func执行的环境。defer是延迟的意思好似伟哥,她把本来要马上发生的f()行为推迟到最后发生。 panic就像二奶一样见光死。
优化点在于 for 循环里不要使用 select + time.After 的组合。
time.After 在压测的时候,我们发现内存占用很高,于是使用 Go Tool PProf 分析 Golang 函数内存申请情况,发现有不断创建 time.After 定时器的问题,定位到是心跳协程里面。
JSON 编解码
开始我们使用官方的 JSON 编解码工具,但由于对性能方面的追求,改为使用滴滴开源的 Json-iterator,使在兼容原生 Golang 的 JSON 编解码工具的同时,效率上有比较明显的提升。以下是压测对比的参考图:
日志模块在起初调研的时候基于性能考虑,确定使用 Uber 开源的 ZAP 库,而且满足业务日志记录的要求。日志库选型很重要,选不好也是影响系统性能和稳定性的。ZAP 的优点包括:
显示代码行号这个需求,ZAP 支持而 Logrus 不支持,这个属于提效的。行号展示对于定位问题很重要。 ZAP 相对于 Logrus 更为高效,体现在写 JSON 格式日志时,没有使用反射,而是用内建的 json encoder,通过明确的类型调用,直接拼接字符串,最小化性能开销。 小坑:每天写一个日志文件的功能,目前 ZAP 不支持,需要自己写代码支持,或者请求系统部支持。
Go 称为云原生时代的母语。「云原生时代,是开发者最好的时代」,在这股浪潮下,我们越早走进 Go,就可能越早在这个新时代抢占关键赛道。
不错的文章Mark, 阿里云高级技术专家探讨 Go 的错误处理
哪些可以 Recover
Recover 最佳实践
对于库如果需要启动 goroutine,如何 recover 呢?
哪些不能 Recover,包括(不限于):
Go1.2 引入了能使用的最多线程数限制ThreadLimit,如果超过了就 panic,这个 panic 是无法 recover 的。
默认是 1 万个物理线程,我们可以调用 runtime 的debug.SetMaxThreads 设置最大线程数。 goroutine 启动时,并不总是新开系统线程,只有当目前所有的物理线程都阻塞在系统调用、cgo 调用,或者显示有调用 runtime.LockOSThread 时。 GOMAXPROCS 只是设置 user-level 并行执行的线程数,也就是真正执行的线程数 。实际上如果物理线程阻塞在 system calls,会开启更多的物理线程。
Concurrent Map Writers,竞争条件,同时写 map,参考下面的例子。推荐使用标准库的 sync.Map 解决这个问题。
Errors in Go,Go 中正确的错误处理,应该是这个例子 Good: CopyFile,虽然啰嗦繁琐不简洁:
package main
import (
"fmt"
"io"
"os"
)
func CopyFile(src, dst string) error {
r, err := os.Open(src)
if err != nil {
return fmt.Errorf("copy %s %s: %v", src, dst, err)
}
defer r.Close()
w, err := os.Create(dst)
if err != nil {
return fmt.Errorf("copy %s %s: %v", src, dst, err)
}
if _, err := io.Copy(w, r); err != nil {
w.Close()
os.Remove(dst)
return fmt.Errorf("copy %s %s: %v", src, dst, err)
}
if err := w.Close(); err != nil {
os.Remove(dst)
return fmt.Errorf("copy %s %s: %v", src, dst, err)
}
return nil
}
func main() {
fmt.Println(CopyFile("src.txt", "dst.txt"))
}
总结下 Go 的 error,错误处理应该注意以下几点:
Best Practice
推荐用github.com/pkg/errors 这个错误处理的库,基本上是够用的,参考 Refine: CopyFile,可以看到 CopyFile 中低级重复的代码已经比较少了:
package main
import (
"fmt"
"github.com/pkg/errors"
"io"
"os"
)
func CopyFile(src, dst string) error {
r, err := os.Open(src)
if err != nil {
return errors.Wrap(err, "open source")
}
defer r.Close()
w, err := os.Create(dst)
if err != nil {
return errors.Wrap(err, "create dest")
}
nn, err := io.Copy(w, r)
if err != nil {
w.Close()
os.Remove(dst)
return errors.Wrap(err, "copy body")
}
if err := w.Close(); err != nil {
os.Remove(dst)
return errors.Wrapf(err, "close dest, nn=%v", nn)
}
return nil
}
func LoadSystem() error {
src, dst := "src.txt", "dst.txt"
if err := CopyFile(src, dst); err != nil {
return errors.WithMessage(err, fmt.Sprintf("load src=%v, dst=%v", src, dst))
}
// Do other jobs.
return nil
}
func main() {
if err := LoadSystem(); err != nil {
fmt.Printf("err %+v\n", err)
}
}
日志的理解
日志是给人看的,是用来查问题的。出现问题后根据某些条件,去查不同进程或服务的日志。日志的关键是不能漏掉信息,漏了关键日志,可能就断了关键的线索;
日志必须要被关联起来,上下文的日志比单个日志更重要。长连接需要根据会话关联日志;不同业务模型有不同的上下文,比如服务器管理把服务器作为关键信息,查询这个服务器的相关日志;全链路跨机器和服务的日志跟踪,需要定义可追踪的逻辑 ID; 常见的上下文包括:
海量日志是给机器看的,是结构化的,能主动报告问题,能从日志中分析潜在的问题。日志的关键是要被不同消费者消费,要输出不同主题的日志,不同的粒度的日志。日志可以用于排查问题,可以用于告警,可以用于分析业务情况。
一些最佳实践:
GO阅读:如何写出优雅的 Go 语言代码
在这篇文章中从三个方面分别介绍了如何写优雅的 Go 语言代码,作者尽可能地给出了最容易操作和最有效的方法:
代码规范:使用辅助工具帮助我们在每次提交 PR 时自动化地对代码进行检查,减少工程师人工审查的工作量;
最佳实践
面向接口:面向接口是 Go 语言鼓励的开发方式,也能够为我们写单元测试提供方便,我们应该遵循固定的模式对外提供功能;
单元测试:保证项目工程质量的最有效办法;
可测试:意味着面向接口编程以及减少单个函数中包含的逻辑,使用『小方法』;
组织方式:使用 Go 语言默认的 Test 框架、开源的 suite 或者 BDD 的风格对单元测试进行合理组织;
Mock 方法:四种不同的单元测试 Mock 方法;
断言:使用社区的 testify 快速验证方法的返回值;
想要写出优雅的代码本身就不是一件容易的事情,它需要我们不断地对自己的知识体系进行更新和优化,推倒之前的经验并对项目持续进行完善和重构,而只有真正经过思考和设计的代码才能够经过时间的检验(代码是需要不断重构的),随意堆砌代码的行为是不能鼓励也不应该发生的,每一行代码都应该按照最高的标准去设计和开发,这是我们保证工程质量的唯一方法。
作者也一直在努力学习如何写出更加优雅的代码,写出好的代码真的不是一件容易的事情,作者也希望能通过这篇文章帮助使用工程师写出更有 Go 语言风格的项目。
图片来自nexus帮助
# docker-compose
version: "3"
services:
nexus:
image: sonatype/nexus3:3.17.0
ports:
- "80:8081"
volumes:
- "./nexus-data:/nexus-data"
athens:
image: gomods/athens:latest
ports:
- "3000:3000"
配置go proxy
从go可执行程序中,查看go的编译版本
$ strings weed | grep "linux.amd64" | sed -n '2,10 p'
/home/travis/.gimme/versions/go1.12.6.linux.amd64
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/container/heap/heap.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/container/ring/ring.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/log/syslog/syslog_unix.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/log/syslog/syslog.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/base32/base32.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/gob/encoder.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/gob/encode.go
/home/travis/.gimme/versions/go1.12.6.linux.amd64/src/encoding/gob/enc_helpers.go
Posted on: 15th April 2019
Filed under: golang tutorial
Occasionally I get asked “why do you like using Go?” And one of the things I often mention is the thoughtful tooling that exists alongside the language as part of the go
command. There are some tools that I use everyday — like go fmt
and go build
— and others like go tool pprof
that I only use to help solve a specific issue. But in all cases I appreciate the fact that they make managing and maintaining my projects easier.
In this post I hope to provide a little background and context about the tools I find most useful, and importantly, explain how they can fit into the workflow of a typical project. I hope it'll give you a good start if you're new to Go.
Or if you've been working with Go for a while, and that stuff's not applicable to you, hopefully you'll still discover a command or flag that you didn't know existed before : )
The information in this post is written for Go 1.12 and assumes that you're working on a project which has modules enabled.
In this post I'll mainly be focusing on tools that are a part of the go
command. But there are a few I'll be mentioning which aren't part of the standard Go 1.12 release.
To install these while using Go 1.12 you'll first need to make sure that you're outside of a module-enabled directory (I usually just change into /tmp
). Then you can use the GO111MODULE=on go get
command to install the tool. For example:
$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/cmd/stress
This will download the relevant package and dependencies, build the executable and add it to your GOBIN
directory. If you haven't explicitly set a GOBIN
directory, then the executable will be added to your GOPATH/bin
folder. Either way, you should make sure that the appropriate directory is on your system path.
Note: This process is a bit clunky and will hopefully improve in future versions of Go. Issue 30515 is tracking the discussion about this.
You can use the go env tool to display information about your current Go operating environment. This can be particularly useful if you're working on an unfamiliar machine.
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/alex/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build245740092=/tmp/go-build -gno-record-gcc-switches"
If there are specific values that you're interested in, you can pass them as arguments to go env
. For example:
$ go env GOPATH GOOS GOARCH
/home/alex/go
linux
amd64
To show documentation for all go env
variables and values you can run:
$ go help environment
During development the go run
tool is a convenient way to try out your code. It's essentially a shortcut that compiles your code, creates an executable binary in your /tmp
directory, and then runs this binary in one step.
$ go run . # Run the package in the current directory
$ go run ./cmd/foo # Run the package in the ./cmd/foo directory
Note: As of Go 1.11 you can pass the path of a package to go run
, like we have above. This means that you no longer have to use workarounds like go run *.go
wildcard expansion to run multiple files. I like this improvement a lot!
Assuming that you've got modules enabled, when you use go run
(or go test
or go build
for that matter) any external dependencies will automatically (and recursively) be downloaded to fulfill the import
statements in your code. By default the latest tagged release of the dependency will be downloaded, or if no tagged releases are available, then the dependency at the latest commit.
If you know in advance that you need a specific version of a dependency (instead of the one that Go would fetch by default) you can use go get
with the relevant version number or commit hash. For example:
$ go get github.com/foo/bar@v1.2.3
$ go get github.com/foo/bar@8e1b8d3
If the dependency being fetched has a go.mod
file, then its dependencies won't be listed in your go.mod
file. In contrast, if the dependency you're downloading doesn't have a go.mod
file, then it's dependencies will be listed in your go.mod
file with an // indirect
comment next to them.
So that means your go.mod
file doesn't necessarily show all the dependencies for your project in one place. Instead, you can view them all using the go list
tool like so:
$ go list -m all
Sometimes you might wonder why is that a dependency? You can answer this with the go mod why command, which will show you the shortest path from a package in your main module to a given dependency. For example:
$ go mod why -m golang.org/x/sys
# golang.org/x/sys
github.com/alexedwards/argon2id
golang.org/x/crypto/argon2
golang.org/x/sys/cpu
Note: The go mod why
command will return an answer for most, but not all, dependencies. Issue 27900 is tracking this.
If you're interested in analyzing or visualizing the dependencies for your application, then you might want to also check out the go mod graph
tool. There's a great tutorial and example code for generating visualizations here.
Lastly, downloaded dependencies are stored in the module cache located at GOPATH/pkg/mod
. If you ever need to clear the module cache you can use the go clean
tool. But be aware: this will remove the downloaded dependencies for all projects on your machine.
$ go clean -modcache
Chances are you're probably familiar with using the gofmt
tool to automatically format your code. But it also supports rewrite rules that you can use to help refactor your code. I'll demonstrate.
Let's say that you have the following code and you want to change the foo
variable to Foo
so it is exported.
var foo int
func bar() {
foo = 1
fmt.Println("foo")
}
To do this you can use gofmt
with the -r
flag to implement a rewrite rule, the -d
flag to display a diff of the changes, and the -w
flag to make the changes in place, like so:
$ gofmt -d -w -r 'foo -> Foo' .
-var foo int
+var Foo int
func bar() {
- foo = 1
+ Foo = 1
fmt.Println("foo")
}
Notice how this is smarter than a find-and-replace? The foo
variable has been changed, but the "foo"
string in the fmt.Println()
statement has been left unchanged. Another thing to note is that the gofmt
command works recursively, so the above command will run on all *.go
files in your current directory and subdirectories.
If you want to use this functionality, I recommend running rewrite rules without the -w
flag first, and checking the diff first to make sure that the changes to the code are what you expect.
Let's take a look at a slightly more complicated example. Say you want to update your code to use the new Go 1.12 strings.ReplaceAll()
function instead of strings.Replace()
. To make this change you can run:
$ gofmt -w -r 'strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)' .
In rewrite rules, single lowercase characters act as wildcards matching arbitrary expressions, and those expressions will be substituted-in in the replacement.
You can view documentation for the standard library packages via your terminal using the go doc
tool. I often use this during development to quickly check something — like the name or signature of a specific function. I find it faster than navigating the web-based documentation and it's always available offline too.
$ go doc strings # View simplified documentation for the strings package
$ go doc -all strings # View full documentation for the strings package
$ go doc strings.Replace # View documentation for the strings.Replace function
$ go doc sql.DB # View documentation for the database/sql.DB type
$ go doc sql.DB.Query # View documentation for the database/sql.DB.Query method
You can also include the -src
flag to display the relevant Go source code. For example:
$ go doc -src strings.Replace # View the source code for the strings.Replace function
You can use the go test
tool to run tests in your project like so:
$ go test . # Run all tests in the current directory
$ go test ./... # Run all tests in the current directory and sub-directories
$ go test ./foo/bar # Run all tests in the ./foo/bar directory
Typically I run my tests with Go's race detector enabled, which can help pick up some of the data races that might occur in real-life usage. Like so:
$ go test -race ./...
It's important to note that enabling the race detector will increase the overall running time of your tests. So if you're running tests very frequently part of a TDD workflow, you might prefer to save using this for a pre-commit test run only.
Since 1.10, Go caches test results at the package-level. If a package hasn't changed between test runs — and you're using the same, cachable, flags for go test
— then the cached test result will be displayed with a "(cached)"
next to it. This is hugely helpful in speeding up the test runtime for large codebases. If you want force your tests to run in full (and avoid the cache) you can use the -count=1
flag, or clear all cached test results by using the go clean
tool.
$ go test -count=1 ./... # Bypass the test cache when running tests
$ go clean -testcache # Delete all cached test results
Note: Cached test results are stored alongside cached build results in your GOCACHE
directory. Check go env GOCACHE
if you're not sure where this is on your machine.
You can limit go test
to running specific tests (and sub-tests) by using the -run
flag. This accepts a regular expression, and only tests which have names that match the regular expression will be run. I like to combine this with the -v
flag to enable verbose mode, so the names of running tests and sub-tests are displayed. It's a useful way to make sure that I haven't screwed up the regexp and that the tests I expect are actually being run!
$ go test -v -run=^TestFooBar$ . # Run the test with the exact name TestFooBar
$ go test -v -run=^TestFoo . # Run tests whose names start with TestFoo
$ go test -v -run=^TestFooBar$/^Baz$ . # Run the Baz subtest of the TestFooBar test only
A couple more flags that it's good to be aware of are -short
(which you can use to skip long-running tests) and -failfast
(which will stop running further tests after the first failure). Note that -failfast
will prevent test results from being cached.
$ go test -short ./... # Skip long running tests
$ go test -failfast ./... # Don't run further tests after a failure.
You can enable coverage analysis when running tests by using the -cover
flag. This will display the percentage of code covered by the tests in the output for each package, similar to this:
$ go test -cover ./...
ok github.com/alexedwards/argon2id 0.467s coverage: 78.6% of statements
You can also generate a coverage profile using the -coverprofile
flag and view it in your web browser by using the go tool cover -html
command like so:
$ go test -coverprofile=/tmp/profile.out ./...
$ go tool cover -html=/tmp/profile.out
This will gives you a navigable listing of all the test files, with code covered by the tests displayed in green, and uncovered code in red.
If you want you can go a step further and set the -covermode=count
flag to make the coverage profile record the exact number of times that each statement is executed during the tests.
$ go test -covermode=count -coverprofile=/tmp/profile.out ./...
$ go tool cover -html=/tmp/profile.out
When viewed in the browser, statements which are executed more frequently are shown in a more saturated shade of green, similar to this:
Note: If you’re using the t.Parallel()
command in any of your tests, then you should use the flag -covermode=atomic
instead of -covermode=count
instead to ensure an accurate count.
Lastly, if you don't have a web browser available to view a coverage profile, you can see a breakdown of test coverage by function/method in your terminal with the command:
$ go tool cover -func=/tmp/profile.out
github.com/alexedwards/argon2id/argon2id.go:77: CreateHash 87.5%
github.com/alexedwards/argon2id/argon2id.go:96: ComparePasswordAndHash 85.7%
...
You can use the go test -count
command to run a test multiple times in succession, which can be useful if you want to check for sporadic or intermittent failures. For example:
$ go test -run=^TestFooBar$ -count=500 .
In this example, the TestFooBar
test will be repeated 500 times in a row. But it's important to note that the test will be repeated in serial — even if it contains a t.Parallel()
instruction. So if your test is doing something relatively slow, like making a round trip to a database, hard disk or the internet, running a large number of tests can take quite a long time.
In that case you might want to use the stress tool to repeat the same test multiple times in parallel instead. You can install it like so:
$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/cmd/stress
To use the stress
tool, you'll first need to compile a test binary for the specific package you want to test. You can do using the go test -c
command. For example, to create a test binary for the package in your current directory:
$ go test -c -o=/tmp/foo.test .
In this example, the test binary will be outputted to /tmp/foo.test
. You can then use the stress
tool to execute a specific test in the test binary like so:
$ stress -p=4 /tmp/foo.test -test.run=^TestFooBar$
60 runs so far, 0 failures
120 runs so far, 0 failures
...
Note: In the example above I've used the -p
flag to restrict the number of parallel processes used by stress
to 4. Without this flag, the tool will default to using a number of processes equal to runtime.NumCPU()
.
Before you build an executable for release or deployment, or distribute your code publicly, you may want to run the go test all
command:
$ go test all
This will run tests on all packages in your module and all dependencies — include testing test dependencies and the necessary standard library packages — and it can help validate that the exact versions of the dependencies being used are compatible with each other. This can take quite a long time to run, but the results cache well so any subsequent tests should be faster in the future. If you want, you could also use go test -short all
to skip any long-running tests.
Go provides two tools to automatically format your code according to the Go conventions: gofmt
and go fmt
. Using these helps keep your code consistent across your files and projects, and — if you use them before committing code — helps reduce noise when examining a diff between file versions.
I like to use the gofmt
tool with the following flags:
$ gofmt -w -s -d foo.go # Format the foo.go file
$ gofmt -w -s -d . # Recursively format all files in the current directory and sub-directories
In these commands, the -w
flag instructs the tool to rewrite files in place, the -s
instructs the tool to apply simplifications to the code where possible, and the -d
flag instructs the tool to output diffs of the changes (because I'm curious to see what is changed). If you want to only display the names of changed files, instead of diffs, you can swap this for the -l
flag instead.
Note: The gofmt
command works recursively. If you pass it a directory like .
or ./cmd/foo
it'll format all .go
files under the directory.
The other formatting tool — go fmt
— tool is a wrapper which essentially calls gofmt -l -w
on a specified file or directory. You can use it like this:
$ go fmt ./...
The go vet
tool carries out static analysis of your code and warns you of things which might be wrong with your code but wouldn't be picked up by the compiler. Issues like unreachable code, unnecessary assignments and badly-formed build tags. You can use it like so:
$ go vet foo.go # Vet the foo.go file
$ go vet . # Vet all files in the current directory
$ go vet ./... # Vet all files in the current directory and sub-directories
$ go vet ./foo/bar # Vet all files in the ./foo/bar directory
Behind the scenes, go vet
runs a bunch of different analyzers which are listed here and you can disable specific ones on a case-by-case basis. For example to disable the composite
analyzer you can use:
$ go vet -composites=false ./...
There are a couple of experimental analyzers in golang.org/x/tools which you might want to try: nilness (which checks for redundant or impossible nil comparisons) and shadow (which check for possible unintended shadowing of variables). If you want to use these, you'll need to install and run them separately. For example, to install nilness
you would run:
$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness
And you can then use it like so:
$ go vet -vettool=$(which nilness) ./...
Note: when the -vettool
flag is used it will only run the specified analyzer — all the other go vet
analyzers won't be run.
As a side note, since Go 1.10 the go test
tool automatically executes a small, high-confidence, subset of the go vet
checks before running any tests. You can turn this behavior off when running tests like so:
$ go test -vet=off ./...
You can use the golint tool to identify style mistakes in your code. Unlike
go vet`, this isn't concerned with correctness of the code, but helps you to align your code with the style conventions in Effective Go and the Go CodeReviewComments.
It's not part of the standard library, so you'll need to install it like so:
$ cd /tmp
$ GO111MODULE=on go get golang.org/x/lint/golint
You can then run it as follows:
$ golint foo.go # Lint the foo.go file
$ golint . # Lint all files in the current directory
$ golint ./... # Lint all files in the current directory and sub-directories
$ golint ./foo/bar # Lint all files in the ./foo/bar directory
Before you commit any changes to your code I recommend running the following two commands to tidy and verify your dependencies:
$ go mod tidy
$ go mod verify
The go mod tidy
command will prune any unused dependencies from your go.mod
and go.sum
files, and update the files to include dependencies for all possible build tags/OS/architecture combinations (note: go run
, go test
, go build
etc are ‘lazy' and will only fetch packages needed for the current build tags/OS/architecture). Running this before each commit will make it easier to determine which of your code changes were responsible for adding or removing which dependencies when looking at the version control history.
I also recommend using the go mod verify
command to check that the dependencies on your computer haven't accidentally (or purposely) been changed since they were downloaded and that they match the cryptographic hashes in your go.sum
file. Running this helps ensure that the dependencies being used are the exact ones that you expect, and any build for that commit will be reproducible at a later point.
To compile a main
package and create an executable binary you can use the go build
tool. Typically I use it in conjunction with the -o
flag, which let's you explicitly set the output directory and name of the binary like so:
$ go build -o=/tmp/foo . # Compile the package in the current directory
$ go build -o=/tmp/foo ./cmd/foo # Compile the package in the ./cmd/foo directory
In these examples, go build
will compile the specified package (and any dependent packages), then invoke the linker to generate an executable binary, and output this to /tmp/foo
.
It's important to note that, as of Go 1.10, the go build
tool caches build output in the build cache. This cached output will be reused again in future builds where appropriate, which can significantly speed up the overall build time. This new caching behavior means that the old maxim of “prefer go install
to go build
to improve caching” no longer applies.
If you're not sure where your build cache is, you can check by running the go env GOCACHE
command:
$ go env GOCACHE
/home/alex/.cache/go-build
Using the build cache comes with one important caveat — it does not detect changes to C libraries imported with cgo
. So if your code imports a C library via cgo
and you've made changes to it since the last build, you'll need to use the -a
flag which forces all packages to be rebuilt. Alternatively, you could use go clean to purge the cache:
$ go build -a -o=/tmp/foo . # Force all packages to be rebuilt
$ go clean -cache # Remove everything from the build cache
Note: Running go clean -cache
will delete cached test results too.
If you're interested in what go build
is doing behind the scenes, you might like to use the following commands:
$ go list -deps . | sort -u # List all packages that are used to build the executable
$ go build -a -x -o=/tmp/foo . # Rebuild everything and show the commands that are run
Finally, if you run go build
on a non-main
package, it will be compiled in a temporary location and again, the result will be stored in the build cache. No executable is produced.
This is one of my favorite features of Go.
By default go build
will output a binary suitable for use on your current operating system and architecture. But it also supports cross-compilation, so you can generate a binary suitable for use on a different machine. This is particularly useful if you're developing on one operating system and deploying on another.
You can specify the operating system and architecture that you want to create the binary for by setting the GOOS
and GOARCH
environment variables respectively. For example:
$ GOOS=linux GOARCH=amd64 go build -o=/tmp/linux_amd64/foo .
$ GOOS=windows GOARCH=amd64 go build -o=/tmp/windows_amd64/foo.exe .
To see a list of all supported OS/architecture combinations you can run go tool dist list
:
$ go tool dist list
aix/ppc64
android/386
android/amd64
android/arm
android/arm64
darwin/386
darwin/amd64
...
Hint: You can use Go's cross-compilation to create WebAssembly binaries.
For a bit more in-depth information about cross compilation I recommend reading this excellent post.
When building your executable you can use the -gcflags
flag to change the behavior of the compiler and see more information about what it's doing. You can see a complete list of available compiler flags by running:
$ go tool compile -help
One flag that you might find interesting is -m
, which triggers the printing of information about optimization decisions made during compilation. You can use it like this:
$ go build -gcflags="-m -m" -o=/tmp/foo . # Print information about optimization decisions
In the above example I used the -m
flag twice to indicate that I want to print decision information two-levels deep. You can get simpler output by using just one.
Also, as of Go 1.10, compiler flags only apply to the specific packages passed to go build
— which in the example above is the package in the current directory (represented by .). If you want to print optimization decisions for all packages including dependencies can use this command instead:
$ go build -gcflags="all=-m" -o=/tmp/foo .
As of Go 1.11, you should find it easier to debug optimized binaries than before. However, you can still use the flags -N
to disable optimizations and -l
to disable inlining if you need to. For example:
$ go build -gcflags="all=-N -l" -o=/tmp/foo . # Disable optimizations and inlining
You can see a list of available linker flags by running:
$ go tool link -help
Probably the most well-known of these is the -X flag, which allows you to "burn in" a (string) value to a specific variable in your application. This is commonly used to add a version number or commit hash. For example:
$ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo .
For more information about the -X
flag and some sample code see this StackOverflow question and this post and this post.
You may also be interested in using the -s
and -w
flags to strip debugging information from the binary. This typically shaves about 25% off the final size. For example:
$ go build -ldflags="-s -w" -o=/tmp/foo . # Strip debug information from the binary
Note: If binary size is something that you need to optimize for you might want to use upx to compress it. See this post for more information.
A nice feature of Go is that it makes it easy to benchmark your code. If you're not familiar with the general process for writing benchmarks there are good guides here and here.
To run benchmarks you'll need to use the go test
tool, with the -bench
flag set to a regular expression that matches the benchmarks you want to execute. For example:
$ go test -bench=. ./... # Run all benchmarks and tests
$ go test -run=^$ -bench=. ./... # Run all benchmarks (and no tests)
$ go test -run=^$ -bench=^BenchmarkFoo$ ./... # Run only the BenchmarkFoo benchmark (and no tests)
I almost always run benchmarks using the -benchmem
flag, which forces memory allocation statistics to be included in the output.
$ go test -bench=. -benchmem ./...
By default, each benchmark test will be run for a minimum of 1 second, once only. You can change this with the -benchtime
and -count
flags:
$ go test -bench=. -benchtime=5s ./... # Run each benchmark test for at least 5 seconds
$ go test -bench=. -benchtime=500x ./... # Run each benchmark test for exactly 500 iterations
$ go test -bench=. -count=3 ./... # Repeat each benchmark test 3 times over
If the code that you're benchmarking uses concurrency, you can use the -cpu
flag to see the performance impact of changing your GOMAXPROCS
value (essentially, the number of OS threads that can execute your Go code simultaneously). For example, to run benchmarks with GOMAXPROCS
set to 1, 4 and 8:
$ go test -bench=. -cpu=1,4,8 ./...
To compare changes between benchmarks you might want to use the benchcmp tool. This isn't part of the standard go
command, so you'll need to install it like so:
$ cd /tmp
$ GO111MODULE=on go get golang.org/x/tools/cmd/benchcmp
You can then use it like this:
$ go test -run=^$ -bench=. -benchmem ./... > /tmp/old.txt
# make changes
$ go test -run=^$ -bench=. -benchmem ./... > /tmp/new.txt
$ benchcmp /tmp/old.txt /tmp/new.txt
benchmark old ns/op new ns/op delta
BenchmarkExample-8 21234 5510 -74.05%
benchmark old allocs new allocs delta
BenchmarkExample-8 17 11 -35.29%
benchmark old bytes new bytes delta
BenchmarkExample-8 8240 3808 -53.79%
Go makes it possible to create diagnostic profiles for CPU use, memory use, goroutine blocking and mutex contention. You can use these to dig a bit deeper and see exactly how your application is using (or waiting on) resources.
There are three ways to generate profiles:
http.DefaultServeMux
which you can then use to generate and download profiles for your running application. This post provides a good explanation and some sample code.pprof.StartCPUProfile()
and pprof.WriteHeapProfile()
functions. See the runtime/pprof documentation for sample code.Or you can generate profiles while running benchmarks or tests by using the various -***profile
flags like so:
$ go test -run=^$ -bench=^BenchmarkFoo$ -cpuprofile=/tmp/cpuprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -memprofile=/tmp/memprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -blockprofile=/tmp/blockprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -mutexprofile=/tmp/mutexprofile.out .
Note: Using the -***profile
flags when running benchmarks or tests will result in a test binary being outputted to your current directory. If you want to output this to an alternative location you should use the -o
flag like so:
$ go test -run=^$ -bench=^BenchmarkFoo$ -o=/tmp/foo.test -cpuprofile=/tmp/cpuprofile.out .
Whichever way you choose to create a profile, when profiling is enabled your Go program will stop about 100 times per second and take a snapshot at that moment in time. These samples are collected together to form a profile that you can analyze using the pprof
tool.
My favorite way to inspect a profile is to use the go tool pprof -http
command to open it in a web browser. For example:
$ go tool pprof -http=:5000 /tmp/cpuprofile.out
This will default to displaying a graph showing the execution tree for the sampled aspects of your application, which makes it possible to quickly get a feel for any resource usage 'hotspots'. In the graph above, we can see that the hotspots in terms of CPU usage are two system calls originating from ioutil.ReadFile()
.
You can also navigate to other views of the profile including top usage by function and source code.
If the amount of information is overwhelming, you might want to use the --nodefraction
flag to ignore nodes that account for less than a certain percentage of samples. For example to ignore nodes that use appear in less than 10% of samples you can run pprof
like so:
$ go tool pprof --nodefraction=0.1 -http=:5000 /tmp/cpuprofile.out
This makes the graph a lot less 'noisy' and if you zoom in on this screenshot, it's now much clearer to see and understand where the CPU usage hotspots are.
Profiling and optimizing resource usage is big, nuanced, topic and I've barely scratched the surface here. If you're interested in knowing more then I encourage you to read the following blog posts:
Another tool that you can use to help diagnose issues is the runtime execution tracer. This gives you a view of how Go is creating and scheduling goroutines to run, when the garbage collector is running, and information about blocking syscall/network/sync operations.
Again, you can generate trace from your tests or benchmarks, or use net/http/pprof
to create and download a trace for your web application. You can then use go tool trace
to view the output in your web browser like so:
$ go test -run=^$ -bench=^BenchmarkFoo$ -trace=/tmp/trace.out .
$ go tool trace /tmp/trace.out
Important: This is currently only viewable in Chrome/Chromium.
For more information about Go's execution tracer and how to interpret the output please see Rhys Hiltner's dotGo 2016 talk and this excellent blog post.
I talked earlier about enabling Go's race detector during tests by using go test -race
. But you can also enable it for running programs when building a executable, like so:
$ go build -race -o=/tmp/foo .
It's critical to note that race-detector-enabled binaries will use more CPU and memory than normal, so you shouldn't use the -race
flag when building binaries for production under normal circumstances.
But you may want to deploy a race-detector-enabled binary on one server within a pool of many. Or use it to help track down a suspected race-condition by using a load-test tool to throw traffic concurrently at a race-detector-enabled binary.
By default, if any races are detected while the binary is running a log will be written to stderr
. You can change this by using the GORACE
environment variable if necessary. For example, to run the binary located at /tmp/foo
and output any race logs to /tmp/race.<pid>
you can use:
$ GORACE="log_path=/tmp/race" /tmp/foo
You can use the go list
tool to check whether a specific dependency has a newer version available like so:
$ go list -m -u github.com/alecthomas/chroma
github.com/alecthomas/chroma v0.6.2 [v0.6.3]
This will output the dependency name and version that you're currently using, followed by the latest version in square brackets [], if a newer one exists. You can also use go list
to check for updates to all dependencies (and sub-dependencies) like so:
$ go list -m -u all
You can upgrade (or downgrade) a dependency to the latest version, specific tagged-release or commit hash with the go get
command like so:
$ go get github.com/foo/bar@latest
$ go get github.com/foo/bar@v1.2.3
$ go get github.com/foo/bar@7e0369f
If the dependency you're updating has a go.mod
file, then based on the information in this go.mod
file, updates to any sub-dependencies will also be downloaded if necessary. If you use the go get -u
flag, the contents of the go.mod
file will be ignored and all sub-dependencies will be upgraded to their latest minor/patch version… even if the go.mod
specifies a different version.
After upgrading or downgrading any dependencies it's a good idea to tidy your modfiles. And you might also want to run the tests for all packages to help check for incompatibilities. Like so:
$ go mod tidy
$ go test all
Occasionally you might want to use a local version of a dependency (for example, you need to use a local fork until a patch is merged upstream). To do this, you can use the go mod edit
command to replace a dependency in your go.mod file with a local version. For example:
$ go mod edit -replace=github.com/alexedwards/argon2id=/home/alex/code/argon2id
This will add a replace rule to your go.mod
file like so, and any future invocations of go run
, go build
etc will use the local version.
File: go.mod
module alexedwards.net/example
go 1.12
require github.com/alexedwards/argon2id v0.0.0-20190109181859-24206601af6c
replace github.com/alexedwards/argon2id => /home/alex/Projects/playground/argon2id
Once it's no longer necessary, you can remove the replace rule with the command:
$ go mod edit -dropreplace=github.com/alexedwards/argon2id
You can use the same general technique to import packages that exist only on your own file system. This can be useful if you're working on multiple modules in development at the same time, one of which depends on the other.
Note: If you don't want to use the go mod edit
command, you can edit your go.mod
file manually to make these changes. Either way will work.
The go fix
tool was originally released back in 2011 (when regular changes were still being made to Go's API) to help users automatically update their old code to be compatible with the latest version of Go. Since then, Go's compatibility promise means if you're upgrading from one Go 1.x version to a newer 1.x version everything should Just Work and using go fix
should generally be unnecessary.
However, there are a handful of very specific issues that it does deal with. You can see a summary of them by running go tool fix -help
. If you decide that you want or need to run go fix
after upgrading, you should you run the following command, then inspect a diff of the changes before you commit them.
$ go fix ./...
If you're confident that you've found an unreported issue with Go's standard library, tooling or documentation, you can use the go bug
command to create a new Github issue.
$ go bug
This will open a browser window containing an issue pre-filled with your system information and reporting template.
Update 2019-04-19: @FedirFR has kindly made a cheatsheet based on this post. You can download it here or go-tooling-cheat-sheet.pdf.
If you enjoyed this blog post, don't forget to check out my new book about how to build professional web applications with Go!
Follow me on Twitter @ajmedwards.
All code snippets in this post are free to use under the MIT Licence.
生产环境pprof,网络不直通(vpn 跳板机),telegraf cpu飙升到200%,咋办
# ps -ef|grep telegraf footsto+ 18552 1 99 2019 ? 101-20:55:59 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d
# top -p 18552 -n 1
top - 15:34:11 up 560 days, 1:29, 1 user, load average: 4.32, 4.32, 4.31
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 41.8 us, 23.6 sy, 0.0 ni, 32.7 id, 0.0 wa, 0.0 hi, 1.8 si, 0.0 st
KiB Mem : 8010196 total, 126864 free, 4718952 used, 3164380 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 2962940 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18552 footsto+ 20 0 117628 59700 4888 S 200.0 0.7 146701:24 telegraf
146701:24 = 146701 minutes and 24 seconds ~= 146701/60/24 ~= 102 days, What does TIME+ in top mean?
Haven't looked at the source code of top, but it seems like 3019:57 in column TIME+ means 3019 minutes 57 seconds of accumulated CPU time.
- 999:00.00 means 999 minutes
- 1000:00 means a thousand minutes (no . separator at all)
I confirmed it on my system. I took an example process from top with 2529:38 then checked it with ps -fp PID which showed 1-18:09:38. The latter is one day and eighteen hours, which is approximately two and a half thousand minutes. Hence it equals the former.
下载安装go工具链linux版本, tar -C /usr/local -xzf go1.13.5.linux-amd64.tar.gz
, export PATH=$PATH:/usr/local/go/bin
telegraf中开启 profiling, telegraf --config telegraf.conf --pprof-addr localhost:6060
Telegraf profiling
if *pprofAddr != "" {
go func() {
pprofHostPort := *pprofAddr
parts := strings.Split(pprofHostPort, ":")
if len(parts) == 2 && parts[0] == "" {
pprofHostPort = fmt.Sprintf("localhost:%s", parts[1])
}
pprofHostPort = "http://" + pprofHostPort + "/debug/pprof"
log.Printf("I! Starting pprof HTTP server at: %s", pprofHostPort)
if err := http.ListenAndServe(*pprofAddr, nil); err != nil {
log.Fatal("E! " + err.Error())
}
}()
}
go tool pprof
开始采集数据
# go tool pprof http://localhost:6060/debug/pprof/profile
Fetching profile over HTTP from http://localhost:6060/debug/pprof/profile
Saved profile in /root/pprof/pprof.telegraf.samples.cpu.001.pb.gz
File: telegraf
Type: cpu
Time: Jan 7, 2020 at 4:31pm (CST)
Duration: 30.09s, Total samples = 1.74s ( 5.78%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 770ms, 44.25% of 1740ms total
Showing top 10 nodes out of 306
flat flat% sum% cum cum%
200ms 11.49% 11.49% 200ms 11.49% runtime.futex
190ms 10.92% 22.41% 230ms 13.22% syscall.Syscall
60ms 3.45% 25.86% 60ms 3.45% runtime.epollwait
60ms 3.45% 29.31% 140ms 8.05% runtime.mapassign_faststr
60ms 3.45% 32.76% 110ms 6.32% runtime.scanobject
50ms 2.87% 35.63% 50ms 2.87% runtime.heapBitsSetType
40ms 2.30% 37.93% 50ms 2.87% runtime.findObject
40ms 2.30% 40.23% 40ms 2.30% runtime.mapiternext
40ms 2.30% 42.53% 40ms 2.30% runtime.memclrNoHeapPointers
30ms 1.72% 44.25% 140ms 8.05% github.com/bjca/telegraf/metric.(*metric).Fields
(pprof)
\# -http=:9100 就是再本地端⼝口9100的http服务上打开分析结果
go tool pprof -http=:9100 http://127.0.0.1:6060/debug/pprof/profile
等待cpu高时,再观察。
参考资料
go tool pprof http://localhost:6060/debug/pprof/profile # 30-second CPU profile go tool pprof http://localhost:6060/debug/pprof/heap # heap profile go tool pprof http://localhost:6060/debug/pprof/block # goroutine blocking profile
使用google/gops,⼀般是在不允许开启http服务,或者应用在容器中
// 应⽤用程序中导⼊入gops代理理
import "github.com/google/gops/agent"
// 非侵⼊式加入代理代码(main.go)
if err := agent.Listen(agent.Options{
ShutdownCleanup: true, // automatically closes on os.Interrupt
}); err != nil {
log.Fatal(err)
}
$ gops pprof-cpu (<pid>|<addr>) \# gops 命令使⽤用收集信息,对应的cpu profile分析命令
$ gops pprof-heap (<pid>|<addr>)
$ gops trace (<pid>|<addr>)
Predefined profiles provided by the runtime/pprof package:
cpu: CPU profile determines where a program spends its time while actively consuming CPU cycles (as opposed to while sleeping or waiting for I/O). heap: Heap profile reports memory allocation samples; used to monitor current and historical memory usage, and to check for memory leaks. threadcreate: Thread creation profile reports the sections of the program that lead the creation of new OS threads. goroutine: Goroutine profile reports the stack traces of all current goroutines. block: Block profile shows where goroutines block waiting on synchronization primitives (including timer channels). Block profile is not enabled by default; use runtime.SetBlockProfileRate to enable it. mutex: Mutex profile reports the lock contentions. When you think your CPU is not fully utilized due to a mutex contention, use this profile. Mutex profile is not enabled by default, see runtime.SetMutexProfileFraction to enable it.
Getting started with Go CPU and memory profiling
package main
import (
//...
"github.com/pkg/profile"
)
func main() {
// CPU profiling by default
defer profile.Start().Stop()
//...
}
2017/08/03 14:26:28 profile: cpu profiling enabled, /var/...../cpu.pprof
go tool pprof --pdf ~/go/bin/yourbinary /var/path/to/cpu.pprof > file.pdf go tool pprof --txt ~/go/bin/yourbinary /var/path/to/cpu.pprof > file.txt
package main
import (
//...
"github.com/pkg/profile"
)
func main() {
// Memory profiling
defer profile.Start(profile.MemProfile).Stop()
//...
}
[High Performance Go Workshop(https://dave.cheney.net/high-performance-go-workshop/sydney-2019.html)
Continuous profiling in Go with Profefe, Profefe
docker run -d -p 10100:10100 profefe/profefe
You can push a profile in profefe:
$ curl -X POST \
"http://localhost:10100/api/0/profiles?service=apid&type=cpu" \
--data-binary @pprof.profefe.samples.cpu.001.pb.gz
{"code":200,"body":{"id":"bo51acqs8snb9srq3p10","type":"cpu","service":"apid","created_at":"2019-12-30T15:18:11.361815452Z"}}
You can retrieve it directly via its ID:
$ go tool pprof http://localhost:10100/api/0/profiles/bo51acqs8snb9srq3p10
Fetching profile over HTTP from http://localhost:10100/api/0/profiles/bo51acqs8snb9srq3p10
Saved profile in /home/gianarb/pprof/pprof.profefe.samples.cpu.002.pb.gz
File: profefe
Type: cpu
Time: Dec 23, 2019 at 4:06pm (CET)
Duration: 30s, Total samples = 0
There is a lot more you can do, when pushing a profile you can set key value pairs called labels and they can be used to query a portion of the profiles.
You can use env=prod|test|dev or region=us|eu and so on.
Retrieving a profile only via ID it’s not the unique way to visualize it. Profefe merges together profiles from the same type in a specific time range:
GET /api/0/profiles/merge?service=<service>&type=<type>&from=<created_from>&to=<created_to>&labels=<key=value,key=value>
It returns the raw compressed binary, it is compatible with go tool pprof as well as the single profile by id.
$ go get golang.org/x/tools/cmd/eg
$ eg -w -t T.template hello
=== /Users/jbd/src/hello/hello.go (1 matches)
Note: There are many .template files underneath the package for testing purposes but they can also be used as a reference how to write other transformation templates.
围棋是一种策略性两人棋类游戏,中国古时称“弈”,西方名称“go”。
Golang多版本安装
method1 官方
method2 gvm
install
bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
influxdb推荐使用gvm来构建: