lzh2nix
commented
4 years ago Go for cloud (2020.08.18)
原文: https://rakyll.org/go-cloud/
列举了为啥golang比较适合云的一些理由。
整个Cloud的生态基本上都是基于golang构建的，期待未来的golang
Open lzh2nix opened 4 years ago
lzh2nix
commented
4 years ago 原文: https://rakyll.org/go-cloud/
列举了为啥golang比较适合云的一些理由。
整个Cloud的生态基本上都是基于golang构建的，期待未来的golang
lzh2nix
commented
4 years ago 原文: https://blog.gopheracademy.com/advent-2016/exposing-go-on-the-internet/
目前的的做法都是将go服务放到nginx后面，然后TLS在nginx这一层终结掉，从nginx到go服务直接走http来降低https带来的性能损耗。这篇文章的建议是直接将go服务暴露在公网上，不过这里纯粹是从性能出发。这篇文章里提到好几个http相关的知识点特别关键。
Server Timeouts
这张图太经典了，将http中的几个timer的关系描述的非常清楚。 对应golang的config
srv := &http.Server{
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 120 * time.Second,
TLSConfig: tlsConfig,
Handler: serveMux,
}Client Timeouts
c := &http.Client{
Transport: &http.Transport{
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
ResponseHeaderTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
}TCP Keepalive 使用ListenAndServe的时候http 默认开启了一个3分钟的tcp keep alive
Mutex 避免将http.DefaultServeMux暴露出去(pprof默认是注册在defaultServeMux上的，如果直接使用default handler的话有可能会把pprof的信息暴露在公网上)，实际项目中使用自己的http.ServeMux.
errror handler
通过 Server.ErrorLog 上std http的错误输出到自定义的logger上
Metrics
使用 Prometheus 对服务进行监控，可以利用 Server.ConnState回调来统计连接的状态(调查leak的问题)
注意
lzh2nix
commented
4 years ago 原文: https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c
golang写http相关的代码简直爽歪，之间接触的都是做个简单的http proxy(就是简单的将所有头和body都进行转发):
func handleHTTP(w http.ResponseWriter, req *http.Request) {
resp, err := http.DefaultTransport.RoundTrip(req)
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
return
}
defer resp.Body.Close()
copyHeader(w.Header(), resp.Header)
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
}
func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}这里的http代理的话就有点特殊，proxy是不知道请求里面的东西的，而且proxy也不做TLS相关的check，这些都转发到正式的server上去。所以这里检测到http Connect 方法之后直接把connection hijacker掉然后，然后做类似转接头的动作。
func handleTunneling(w http.ResponseWriter, r *http.Request) {
dest_conn, err := net.DialTimeout("tcp", r.Host, 10*time.Second)
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
hijacker, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "Hijacking not supported", http.StatusInternalServerError)
return
}
client_conn, _, err := hijacker.Hijack()
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
}
go transfer(dest_conn, client_conn)
go transfer(client_conn, dest_conn)
}
func transfer(destination io.WriteCloser, source io.ReadCloser) {
defer destination.Close()
defer source.Close()
io.Copy(destination, source)
}原文: https://nanxiao.me/en/a-brief-intro-of-tcp-keep-alive-in-gos-http-implementation/
一篇简单介绍golang http底层connect的文章， 不过这篇文章很明显从外部来分析的，对源码的分析还是不够深入，其实这里对socket四元组有一个简单的LRU cache，详细的代码可以参考之前我http client源码的分析
lzh2nix
commented
4 years ago 原文: https://eli.thegreenplace.net/2019/go-json-cookbook/
确实是goalng的操作json的cookbook(:smirk:,:smirk:,:smirk:),包含了json相关的所有知识点
json:"id,omitempty" marshal的时候如果为零值就忽略json:"-" 不对该字段进行marshal
struct Event {
Name string `json:"name"`
Url string `json:"url"`
}func main() {
bb := []byte( { "event": {"name": "joe", "url": "event://101"}, "otherstuff": 15.2, "anotherstuff": 100 })
var m map[string]json.RawMessage
if err := json.Unmarshal(bb, &m); err != nil {
panic(err)
}
if eventRaw, ok := m["event"]; ok {
var event Event
if err := json.Unmarshal(eventRaw, &event); err != nil {
panic(err)
}
fmt.Println("Parsed Event:", event)
} else {
fmt.Println("Can't find 'event' key in JSON")
}}
- 如果是一个Pointer类型变量，如果没有初始化，marshal时就会是使用零值(nil for pointer)
- 通过实现以下接口来实现自定义的marshal
``` golang
type Marshaler interface {
MarshalJSON() ([]byte, error)
}type Account struct {
Id int32
Name string
}
func (a Account) MarshalJSON() ([]byte, error) {
m := map[string]string{
"id": fmt.Sprintf("0x%08x", a.Id),
"name": a.Name,
}
return json.Marshal(m)
}官方json库不一定是最好的实现 选择最合适的json库
https://github.com/json-iterator/go https://medium.com/a-journey-with-go/go-is-the-encoding-json-package-really-slow-62b64d54b148
json 转 golang 结构体在线工具: https://mholt.github.io/json-to-go/
lzh2nix
commented
4 years ago 原文:http://mmcloughlin.com/posts/your-pprof-is-showing
本文主要讲了怎样避免将你的pprof暴露在公网上。
package main
import (
"fmt"
"log"
"net/http"
_ "net/http/pprof" // here be dragons
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello World!")
})
log.Fatal(http.ListenAndServe(":8080", nil))
}这种模式下大家都注册在了默认的DefaultServeMux上，以下的router都会暴露出去:
/debug/pprof/profile: 30-second CPU profile /debug/pprof/heap: heap profile /debug/pprof/goroutine?debug=1: all goroutines with stack traces /debug/pprof/trace: take a trace / Hello world
解决方式就是将 将pprof和业务分开使用两个ServeMux， pprof listen在内网地址
// Pprof server.
go func() {
log.Fatal(http.ListenAndServe("localhost:8081", nil))
}()
// Application server.
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello World!")
})
log.Fatal(http.ListenAndServe(":8080", mux))原文: https://blog.questionable.services/article/map-string-interface/
主要围绕着官方context之前的context
type contextKey int// Define keys that support equality. const csrfKey contextKey = 0 const userKey contextKey = 1
func GetCSRFToken(r *http.Request) (string, error) { val, ok := context.GetOk(r, csrfKey) // 获取 return token, nil }
func CSRFMiddleware(h http.Handler) http.Handler { return func(w http.ResponseWriter, r *http.Request) { context.Set(r, csrfKey, token) // 设置 h.ServeHTTP(w, r) } }
- goji的per request模式(每个request都有一个env)
```golang
func GetContextString(c web.C, key string) (string, error) {
val, ok := c.Env[key].(string)
if !ok {
return "", ErrTypeNotPresent
}
return val, nil
}import ( "fmt" "log" "net/http"
"github.com/gocraft/web")
type Context struct { CSRFToken string User string }
// Our middleware and handlers must be defined as methods on our context struct, // or accept the type as their first argument. This ties handlers/middlewares to our // particular application structure/design. func (c Context) CSRFMiddleware(w web.ResponseWriter, r web.Request, next web.NextMiddlewareFunc) { token, err := GenerateToken(r) if err != nil { http.Error(w, "No good!", http.StatusInternalServerError) return }
c.CSRFToken = token
next(w, r)}
func (c Context) ShowSignupForm(w web.ResponseWriter, r web.Request) { // No need to type assert it: we know the type. // We can just use the value directly. fmt.Fprintf(w, "Our token is %v", c.CSRFToken) }
func main() { router := web.New(Context{}).Middleware((Context).CSRFMiddleware) router.Get("/signup", (Context).ShowSignupForm)
err := http.ListenAndServe(":8000", router)
if err != nil {
log.Fatal(err)
}}
官方的context其实和第二种有点类似原文: https://rollout.io/blog/using-object-oriented-web-servers-go/
简单demo性质的http server可以直接挂载到 http.defaultServeMux 上，复杂的工程的话还是建议用其他框架或者用类似下面的面向对象式的web server。
func main() {
http.ListenAndServe(":8080", New())
}
func New() http.Handler {
mux := http.NewServeMux()
log := log.New(os.Stdout, "web ", log.LstdFlags)
app := &app{mux, log}
mux.HandleFunc("/foo", app.foo)
return app
}
type app struct {
mux *http.ServeMux
log *log.Logger
}
func (a *app) ServeHTTP(w http.ResponseWriter, r *http.Request) {
a.mux.ServeHTTP(w, r)
}
func (a *app) foo(w http.ResponseWriter, r *http.Request) {
a.log.Println("request to foo")
}这块可以参考下我司的http框架， 面向对象这块儿确实做很好https://github.com/qiniu/http/blob/master/examples/baserestrpc/restrpc_example.go
lzh2nix
commented
4 years ago 原文: https://pliutau.com/handle-http-request-errors-in-go/
关于http错误有个血的教训就是千万不要用http code来表示错误码， 错误码保持唯一然后放到body里。下面是我司的处理方式:
package main
import (
"net/http"
"github.com/qiniu/http/httputil"
)
var (
errorBadUserName = httputil.NewErrorEx(400, 400001, "invalid arguments(bad usename)")
)
func main() {
http.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {
httputil.Error(w, errorBadUserName)
})
http.ListenAndServe(":8080", nil)
}错误提示如下：
➜ http curl http://127.0.0.1:8080/register | jq
{
"error": "invalid arguments(bad usename)",
"errno": 400001
}具体 httputil 可以参考
https://github.com/qiniu/http/blob/master/httputil/httputil.go
lzh2nix
commented
4 years ago 一篇很好的介绍gRPC+Protocol Buffers的文章，在ngnix 1.13中宣布直接支持gRPC访问了(https://www.nginx.com/blog/nginx-1-13-10-grpc/), 所以未来gRPC可能不止服务间的通信，有可能往外延伸，可以作为B/S之间通信的主要方式。 作者在另外一片文章中也给出里各种序列的性能对比 https://medium.com/@shijuvar/benchmarking-protocol-buffers-json-and-xml-in-go-57fa89b8525：
从这个测试结果来看的话gRPC具有很大的优势。
lzh2nix
commented
4 years ago 原文: https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779
这个思想在之前的文章中也提到过，主要的原因就是defaultClient中默认超时时间都是0,也就是永不超时，如果线上出问题查起来就很痛苦(从日志/监控完全看不出来，估计也就抓包能够分析出来)。
建议的方式：
var netTransport = &http.Transport{
Dial: (&net.Dialer{
Timeout: 5 * time.Second,
}).Dial,
TLSHandshakeTimeout: 5 * time.Second,
}
var netClient = &http.Client{
Timeout: time.Second * 10,
Transport: netTransport,
}原文: https://blog.gopheracademy.com/advent-2016/http-client/
使用第三方服务的时候，如果它提供了sdk的话就最好了，如果没有提供的话就面临着自己需要写一个client，这篇文章也是接着上一篇的思路，在这篇文章中作者建议是使用context.WithDeadline来处理超时问题
req, err := http.NewRequest(method, url, body)
req = req.WithContext(ctx)
return client.Do(req)type NullTime struct {
Time time.Time
Valid bool
}
func (nt *NullTime) UnmarshalJSON(b []byte) error {
if string(b) == "null" {
nt.Valid = false
return nil
}
var t time.Time
err := json.Unmarshal(b, &t)
if err != nil {
return err
}
nt.Valid = true
nt.Time = t
return nil
}建议user agent包含一下几个东西：
twilio-go/0.54 rest-client/0.16 (https://github.com/kevinburke/rest) go1.7.4 (darwin/amd64)这个是最容易忽略的一点，在定义函数的时候很容易将每个参数独立放到参数列表里，这样很不容易扩展，如果额外加了一个参数就很尴尬， 如果是GET请求的话建议将参数放到url.Value{}, 其他请求直接传递一个struct
data := url.Values{}
data.Set("id", "14105551234")
client.Calls.queryDevice(data)可以使用interface/写一个httptest.NewServer 方式测试第三方api
type ChargeResource interface {
Get(string) (*Charge, error)
Create(url.Values) (*Charge, error)
List(url.Values) ([]*Charge, error)
}
vs
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(400)
w.Write([]byte(`{"message": "Card charge denied", "code": 10002}`))
}))
defer s.Close()
client.Base = s.URL
charge, err := client.Charge.Create(...)
if err == nil {
t.Fatal("Expected to get error...")
}原文: hhttps://pace.dev/blog/2018/05/09/how-I-write-http-services-after-eight-years.html
这篇可以作为整个web章节的总结了
type server struct {
db *someDatabase
router *someRouter
email EmailSender
}func (s *server) routes() {
s.router.HandleFunc("/api/", s.handleAPI())
s.router.HandleFunc("/about", s.handleAbout())
s.router.HandleFunc("/", s.handleIndex())
}
func (s *server) handleSomething() http.HandlerFunc {
thing := prepareThing()
return func(w http.ResponseWriter, r *http.Request) {
// use thing
}
}func (s server) handleSomething() http.HandlerFunc {
thing := prepareThing()
return func(w http.ResponseWriter, r http.Request) {
// use thing
}
}
- 4. 中间件的实现
```golang
func (s *server) adminOnly(h http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if !currentUser(r).IsAdmin {
http.NotFound(w, r)
return
}
h(w, r)
}
}
func (s *server) routes() {
s.router.HandleFunc("/api/", s.handleAPI())
s.router.HandleFunc("/about", s.handleAbout())
s.router.HandleFunc("/", s.handleIndex())
s.router.HandleFunc("/admin", s.adminOnly(s.handleAdminIndex()))
}func TestGreet(t *testing.T) {
is := is.New(t)
p := struct {
Name string `json:"name"`
}{
Name: "Mat Ryer",
}
var buf bytes.Buffer
err := json.NewEncoder(&buf).Encode(p)
is.NoErr(err) // json.NewEncoder
req, err := http.NewRequest(http.MethodPost, "/greet", &buf)
is.NoErr(err)
//... more test code herefunc TestHandleAbout(t *testing.T) {
is := is.New(t)
srv := server{
db: mockDatabase,
email: mockEmailSender,
}
srv.routes()
req := httptest.NewRequest("GET", "/about", nil)
w := httptest.NewRecorder()
srv.ServeHTTP(w, req)
is.Equal(w.StatusCode, http.StatusOK)
}
目录
Go for Cloud(2020.08.18) Exposing Go on the Internet(2020.08.19) HTTP(S) Proxy in Golang in less than 100 lines of code(2020.08.19) A brief intro of TCP keep-alive in Go’s HTTP implementation(2020.08.20) Go JSON Cookbook(2020.08.20) Your pprof is showing: IPv4 scans reveal exposed net/http/pprof endpoints(2020.08.21) HTTP Request Contexts & Go(2020.08.21) Using Object-Oriented Web Servers in Go(2020.08.22) Handle HTTP Request Errors in Go(2020.08.22) Building High Performance APIs In Go Using gRPC And Protocol Buffers(2020.08.24) Don't use Go's default HTTP client (in production)(2020.08.24) Writing an API Client in Go(2020.08.25) How I write Go HTTP services after seven years(2020.08.25)