bingoohuang commented 4 years ago

Continuous Benchmark for Go Project

cob compares benchmarks between the latest commit (HEAD) and the previous commit (HEAD{@1}). The program will fail if the change in score is worse than the threshold. This tools is suitable for CI/CD to detect a regression of a performance automatically.

Go package to make lightweight ASCII line graphs ╭┈╯.

func main() {
    data := []float64{3, 4, 9, 6, 2, 4, 5, 8, 5, 10, 2, 7, 2, 5, 6}
    graph := asciigraph.Plot(data)

    fmt.Println(graph)
}

seq 1 72 | asciigraph -h 10 -c "plot data from stdin"
 72.00 ┼                                                                  ╭────
 64.90 ┤                                                           ╭──────╯
 57.80 ┤                                                    ╭──────╯
 50.70 ┤                                             ╭──────╯
 43.60 ┤                                      ╭──────╯
 36.50 ┤                              ╭───────╯
 29.40 ┤                       ╭──────╯
 22.30 ┤                ╭──────╯
 15.20 ┤         ╭──────╯
  8.10 ┤  ╭──────╯
  1.00 ┼──╯
          plot data from stdin

sqlmw provides an absurdly simple API that allows a caller to wrap a database/sql driver with middleware.

func run(dsn string) {
        // install the wrapped driver
        sql.Register("postgres-mw", sqlmw.Driver(pq.Driver{}, new(sqlInterceptor)))
        db, err := sql.Open("postgres-mw", dsn)
        ...
}

type sqlInterceptor struct {
        sqlmw.NullInterceptor
}

func (in *sqlInterceptor) StmtQueryContext(ctx context.Context, conn driver.StmtQueryContext, query string, args []driver.NamedValue) (driver.Rows, error) {
        startedAt := time.Now()
        rows, err := conn.QueryContext(ctx, args)
        log.Debug("executed sql query", "duration", time.Since(startedAt), "query", query, "args", args, "err", err)
        return rows, err
}

// Logging
func (in *sqlInterceptor) StmtQueryContext(ctx context.Context, conn driver.StmtQueryContext, query string, args []driver.NamedValue) (driver.Rows, error) {
        startedAt := time.Now()
        rows, err := conn.QueryContext(ctx, args)
        log.Debug("executed sql query", "duration", time.Since(startedAt), "query", query, "args", args, "err", err)
        return rows, err
}
// Tracing
func (in *sqlInterceptor) StmtQueryContext(ctx context.Context, conn driver.StmtQueryContext, query string, args []driver.NamedValue) (driver.Rows, error) {
        span := trace.FromContext(ctx).NewSpan(ctx, "StmtQueryContext")
        span.Tags["query"] = query
        defer span.Finish()
        rows, err := conn.QueryContext(ctx, args)
        if err != nil {
                span.Error(err)
        }
        return rows, err
}
// Retries
func (in *sqlInterceptor) StmtQueryContext(ctx context.Context, conn driver.StmtQueryContext, query string, args []driver.NamedValue) (driver.Rows, error) {
        for {
                rows, err := conn.QueryContext(ctx, args)
                if err == nil {
                        return rows, nil
                }
                if err != nil && !isIdempotent(query) {
                        return nil, err
                }
                select {
                case <-ctx.Done():
                        return nil, ctx.Err()
                case <-time.After(time.Second):
                }
        }
}

bingoohuang commented 4 years ago

Casbin

Golang最强大的访问控制框架casbin全解析

控制访问模型有哪几种？我们需要先来了解下这个。

UGO(User, Group, Other) 这个是Linux中对于资源进行权限管理的访问模型。Linux中一切资源都是文件，每个文件都可以设置三种角色的访问权限（文件创建者，文件创建者所在组，其他人）。这种访问模型的缺点很明显，只能为一类用户设置权限，如果这类用户中有特殊的人，那么它无能为力了。
ACL(访问控制列表) 它的原理是，每个资源都配置有一个列表，这个列表记录哪些用户可以对这项资源进行CRUD操作。当系统试图访问这项资源的时候，会首先检查这个列表中是否有关于当前用户的访问权限，从而确定这个用户是否有权限访问当前资源。linux在UGO之外，也增加了这个功能。
RBAC(基于角色的权限访问控制) 这个是很多业务系统最通用的权限访问控制系统。它的特点是在用户和具体权限之间增加了一个角色。就是先设置一个角色，比如管理员，然后将用户关联某个角色中，再将角色设置某个权限。用户和角色是多对多关系，角色和权限是多对多关系。所以一个用户是否有某个权限，根据用户属于哪些角色，再根据角色是否拥有某个权限来判断这个用户是否有某个权限。

RBAC的逻辑有更多的变种。
- 变种一：角色引入继承
  
  角色引入了继承概念，那么继承的角色有了上下级或者等级关系。
- 变种二：角色引入了约束
  
  角色引入了约束概念。约束概念有两种，
  - 一种是静态职责分离：
    - 互斥角色：同一个用户在两个互斥角色中只能选择一个
    - 基数约束：一个用户拥有的角色是有限的，一个角色拥有的许可也是有限的
    - 先决条件约束：用户想要获得高级角色，首先必须拥有低级角色
  - 一种是动态职责分离：可以动态的约束用户拥有的角色，如一个用户可以拥有两个角色，但是运行时只能激活一个角色。
- 变种三：既有角色约束，又有角色继承, 就是前面两种角色变种的集合。

ABAC(基于属性的权限验证) Attribute-based access control，这种权限验证模式是用属性来标记资源权限的。比如k8s中就用到这个权限验证方法。比如某个资源有pod属性，有命名空间属性，那么我设置的时候可以这样设置： Bob可以在命名空间projectCaribou 中读取 pod：
```
{
  "apiVersion": "abac.authorization.kubernetes.io/v1beta1",
  "kind": "Policy",
  "spec": {
    "user": "bob",
    "namespace": "projectCaribou",
    "resource": "pods",
    "readonly": true
  }
}
```
这个权限验证模型的好处就是扩展性好，一旦要增加某种权限，就可以直接增加某种属性。
DAC（自主访问控制）在ACL的访问控制模式下，有个问题，能给资源增加访问控制的是谁，这里就有几种办法，比如增加一个super user，这个超级管理员来做统一的操作。还有一种办法，有某个权限的用户来负责给其他用户分配权限。这个就叫做自主访问控制。
MAC(强制访问控制) 强制访问控制和DAC相反，它不将某些权限下放给用户，而是在更高维度（比如操作系统）上将所有的用户设置某些策略，这些策略是需要所有用户强制执行的。这种访问控制也是基于某些安全因素考虑。
PML casbin 是一种典型的“配置即一切”的软件思路，那么它的配置语法就显得格外重要。我们可以通过 casbin 的在线配置编辑器来进行学习。 PML(PERM modeling language)。其中的 PERM 指的是 Policy-Effect-Request-Matcher 。

bingoohuang commented 4 years ago

evaluating arbitrary C-like artithmetic/string expressions.

expression, err := govaluate.NewEvaluableExpression("(requests_made * requests_succeeded / 100) >= 90");

parameters := make(map[string]interface{}, 8)
parameters["requests_made"] = 100;
parameters["requests_succeeded"] = 80;

result, err := expression.Evaluate(parameters);
// result is now set to "false", the bool value.

bingoohuang commented 4 years ago

dns server with redis backend How we optimized our DNS server using go tools

bingoohuang commented 4 years ago

要看一下网卡的速度，正常是这样子的

[root@fs04-192-168-126-5 ~]# ethtool eno1 |grep -i speed
    Speed: 1000Mb/s
[root@fs04-192-168-126-5 ~]# cat /sys/class/net/eno1/speed
1000

可是阿里云ECS上是这样子的，就是看不了咯

[footstone@apigateway01 ~]$ ethtool eth0
Settings for eth0:
Cannot get wake-on-lan settings: Operation not permitted
    Link detected: yes
[footstone@apigateway01 ~]$ cat /sys/class/net/eth0/speed
cat: /sys/class/net/eth0/speed: Invalid argument

看来只能用测试工具自己测一下了。

Ethr：一款TCP、UDP和HTTP网络性能测量工具

Ethr是一个用golang编写的跨平台网络性能测量工具。该项目的目标是提供本机工具，用于跨多种协议（如TCP，UDP，HTTP，HTTPS和跨多个平台）对带宽，连接，数据包，延迟，丢失进行全面的网络性能测量。

footstone@apigateway01 ~]$ ./ethr -s -ports control=9888,tcp=9999,udp=9999,http=9899,https=9799 -4
Listening on 9999 for TCP bandwidth tests
Listening on 9998 for TCP conn/s tests
Listening on 9996 for TCP latency tests
Listening on 9899 for HTTP bandwidth tests
Listening on 9799 for HTTPS bandwidth tests
Listening on 9888 for control plane
New control connection from 192.168.37.82, port 41136
Starting TCP Bandwidth test from 192.168.37.82
-----------------------------------------------------------
[RemoteAddress]  Proto   Bits/s   Conn/s    Pkt/s   Latency
[192.168.37.82]    TCP    2.03G
[192.168.37.82]    TCP    1.06G
[192.168.37.82]    TCP    1.05G
[192.168.37.82]    TCP    1.05G
[192.168.37.82]    TCP  968.96M
[192.168.37.82]    TCP    1.12G
[192.168.37.82]    TCP    1.06G
[192.168.37.82]    TCP    1.06G
[192.168.37.82]    TCP    1.03G
[192.168.37.82]    TCP    1.06G
Ending Bandwidth test from 192.168.37.82

[footstone@apigateway02 ~]$ ./ethr -ports control=9888,tcp=9999,udp=9999,http=9899,https=9799 -4 -c 192.168.29.11 -r
Connecting to host [192.168.29.11], port 9999
[  6] local 192.168.37.82 port 51788 connected to 192.168.29.11 port 9999
- - - - - - - - - - - - - - - - - - - - - - -
[ ID]   Protocol    Interval      Bits/s
[  6]     TCP      000-001 sec     2.01G
[  6]     TCP      001-002 sec     1.06G
[  6]     TCP      002-003 sec     1.06G
[  6]     TCP      003-004 sec     1.05G
[  6]     TCP      004-005 sec   971.26M
[  6]     TCP      005-006 sec     1.11G
[  6]     TCP      006-007 sec     1.06G
[  6]     TCP      007-008 sec     1.06G
[  6]     TCP      008-009 sec     1.04G
[  6]     TCP      009-010 sec     1.06G
Ethr done, duration: 10s.

查看了一下ethr的计量单位，都是以1000为除数，因此测试出来的1.06G bps 相当于千兆网卡的速度.

const (
    // UNO represents 1 unit.
    UNO = 1

    // KILO represents k.
    KILO = 1000

    // MEGA represents m.
    MEGA = 1000 * 1000

    // GIGA represents g.
    GIGA = 1000 * 1000 * 1000

    // TERA represents t.
    TERA = 1000 * 1000 * 1000 * 1000
)

bingoohuang commented 4 years ago

OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

bingoohuang commented 4 years ago

GO GUI

Cross platform GUI in Go based on Material Design

bingoohuang commented 4 years ago

工具：

mgechev/revive: 🔥 ~6x faster, stricter, configurable, extensible, and beautiful drop-in replacement for golint.

bingoohuang commented 4 years ago

const json = `{"name":{"first":"Janet","last":"Prichard"},"age":47}`

func main() {
    value := gjson.Get(json, "name.last")
    println(value.String())
}

Simdjson-Go: 在 Go 中每秒解析 JSON 的千兆字节，介绍

由丹尼尔 · 莱米尔和杰夫 · 朗戴尔设计的Simdjson使用了一种新颖的两阶段方法，通过这种方法可以在单核上实现每秒千兆字节的 JSON 解析性能。利用所谓的 SIMD 指令，可以根据每条指令执行更多的文本和数字运算，并显著提高性能。

bingoohuang commented 4 years ago

中文版awesome-go

bingoohuang commented 4 years ago

GoAWK: an AWK interpreter written in Go
unidoc.io/unioffice Pure go library for creating and processing Office Word (.docx), Excel (.xlsx) and Powerpoint (.pptx) documents
Retry, Race, All, Some, etc strategies for http.Client calls
Simple, secure and modern Go HTTP server to serve static sites, single-page applications or a file with ease
An HTTP performance testing tool written in GoLang
netscanner - TCP/UDP scanner to find open or closed ports

https://github.com/checkr/flagr

Introducing Flagr: A robust, high-performance service for feature flagging and A/B testing

Architecture

There are three components: Flagr Evaluator, Flagr Manager, and Flagr Metrics.

Rules engine and user segmentation

The API is written in Go-Swagger. We’ve considered gRPC, Thrift, gRPC-gateway and plain REST, and the Go implementation of swagger really stands out here in terms of the popularity of REST API and the client code generation support.

REST API (Built with Go-Swagger)

Debug Console (test your flag evaluation)

Editing History

Kafka Data Logging & JWT Auth User login (via env variable configuration)

bingoohuang commented 4 years ago

urfave/negroni 面向 Golang 的惯用 HTTP 中间件 Classic ()提供了一些对大多数应用程序有用的默认中间件:
- negroni.Recovery - Panic Recovery Middleware. - 应急恢复中间件
- negroni.Logger - Request/Response Logger Middleware. - 请求 / 响应记录器中间件
- negroni.Static - Static File serving under the "public" directory. - ”公共”目录下的静态文件
- 还有众多的第三方插件
- gin and fresh both live reload negroni apps.
jiacai2050/prosumer A producer-consumer solution for Golang.

bingoohuang commented 4 years ago

Pogreb is an embedded key-value store for read-heavy workloads written in Go.

https://github.com/akrylysov/pogreb

bingoohuang / blog

golang库跟踪 #141

Casbin