Prometheus based metrics

[arpitjindal97]

Can we have an option to expose prometheus metrics ? Similar to what envoy exposes

You can take a look at them by running envoy server locally. Use below code and commands to have a working envoy routing to a microservice.

Run with:

docker run -it --rm -v ${PWD}/envoy.yaml:/etc/envoy/envoy.yaml -p 10000:10000 -p 8000:8000 envoyproxy/envoy-dev:latest

Visit: localhost:8000/stats/prometheus

envoy.yaml

node:
  cluster: front-envoy

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          tracing:
            provider:
              name: envoy.tracers.zipkin
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
                collector_cluster: jaeger
                collector_endpoint: "/api/v2/spans"
                shared_span_context: false
                collector_endpoint_version: HTTP_JSON
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service_envoyproxy_io
  clusters:
  - name: service_envoyproxy_io
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: service_envoyproxy_io
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.19
                port_value: 8080
  - name: jaeger
    connect_timeout: 1s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: jaeger
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 192.168.1.19
                port_value: 9411

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8000

main.go

package main

import (
    "io"
    "log"
    "net/http"
)

func main() {
    // Set routing rules
    http.HandleFunc("/endpoint1", endpoint1)
    http.HandleFunc("/endpoint2", endpoint2)

    //Use the default DefaultServeMux.
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        log.Fatal(err)
    }
}

func endpoint1(w http.ResponseWriter, r *http.Request) {
    io.WriteString(w, "Endpoint 1")
}

func endpoint2(w http.ResponseWriter, r *http.Request) {
    io.WriteString(w, "Endpoint 2")
}

[xgfone]

Yes, i understood.

Envoy is a completed product, especially a binary program, which needs a way to expose the inner information, especially running state.

But, go-apiserver is only a framework or library, and used to implement a binary program. So you have lots of ways to expose what you want to publish. When designing the API interface, it supports CRUD, such as GetXXX or GetAllXXX to get the inner data. At the meanwhile, it also supports the unified and consistent middleware interface to intercept the request or response. So you maybe use the middleware to collect the statistics and expose them. It is difficult to determine what metrics to expose because business or demand is unpredictable.

[arpitjindal97]

Following middleware, one more idea came to my mind that is, we can have a middleware library that can be plugged into the code. It will already have code to collect stats/metrics and expose them. What do you think ? Can be an idea for your next repo

That library can further be extended to provide support for tracing.

Additionally, It can expose metrics in various formats to be compatible with different scrapers

Metrics:

• Prometheus
• statsd
• OpenMetrics.io

Tracing:

• Zipkin
• Jaeger
• opentracing.io

[xgfone]

Yes, you're right. I'm thinking of supporting OpenTelemetry to implement Metric and Tracing. OpenTracing is deprecated and OpenTelemetry is the substitute.

I hope that go-apiserver keeps unsophisticated and the minimized third-party dependencies. If implementing these middlewares, therefore, i maybe open a new repository, or put them into the middleware collection repository go-http-middlewares, which has provided a middleware based on prometheus.

If you has a good idea. give me?

[arpitjindal97]

I would say put them in go-http-middlewares repo which you already have. It should be plug-n-play.

Anyone who wants to expose metrics should just add a line to his code and automatically relevant metrics should be exposed on a configurable endpoint default: /metrics.

What metrics to expose is always a challenge. Starting off with some basic metrics will be a good idea and later more can be added as more people start using it (see envoy metrics for ref)

I see you already have something for prometheus, my suggestion is to put these functionalities in that first. Adding more standards should be relatively easy, can be done later.

[xgfone]

I'm with you on that.

[arpitjindal97]

BTW, how is ship different from go-apiserver ?

[xgfone]

ship is only an echo-like http router, not more. And you can consider it as another echo.

go-apiserver is more, not only a http router, such as HTTP LoadBalancer, Virtual Host, TLS Certificate, Data Validation, TCP on TLS, etc. it is much easier to implement an API gateway by using go-apiserver, see Mini API Gateway. For the data, on the management side, most of components in go-apiserver support CRUD, which is thread-safe; on the forwarding side, such as routing the request, it's atomic, no lock. Between management and forwarding, go-apiserver uses data redundancy to implement it.

For the http router, moreover, go-apiserver is more flexible and powerful than ship or echo. All of gin, ship and echo use Radix tree to implement the router. But go-apiserver is based on the rule, that's, build a route by the rule. So it may do more. See ruler. go-apiserver router supports the ship-like Conext handler and http.Handler to handle the request.

When using ship, I hope that my web simultaneously supports HTTP and HTTPS and selects HTTP or HTTPS by the client request, see TLS Certificate. When developing an API gateway, it needs to support CRUD, but Radix tree is too rigid. So I decide to develop a new framework or library, that's, go-apiserver.

[xgfone]

go-apiserver uses http.Handler as the standard http request handler. So, for the middleware, it also uses http.Handler, and you can use http.Handler to create the http middleware, for example,

middleware.NewMiddleware("name", priority, func(h interface{}) interface{} {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request){
        // TODO
        h.(http.Handler).ServeHTTP(w, r)
        // TODO
    })
}),

OpenTelemetry has provided the official adapter for http.Transport and http.Handler. So we can use them directly, For example

http.DefaultClient = &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

middleware.NewMiddleware("otel", 100, func(h interface{}) interface{} {
    return otelhttp.NewHandler(h.(http.Handler), "HTTPServerName")
})

So what we only need to do is to install the exporter, for example

var defaultResource *resource.Resource

func installPrometheusAsMetricExporter() {
    factory := processor.NewFactory(
        selector.NewWithHistogramDistribution(),
        aggregation.CumulativeTemporalitySelector(),
    )

    ctrl := controller.New(factory, controller.WithResource(defaultResource))
    exporter, err := prometheus.New(prometheus.Config{}, ctrl)
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    ruler.DefaultRouter.Path("/metrics").GET(exporter)
    global.SetMeterProvider(exporter.MeterProvider())
}

func installJaegerAsTracerExporter() {
    exporter, err := jaeger.New(jaeger.WithAgentEndpoint())
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    otel.SetTracerProvider(trace.NewTracerProvider(
        trace.WithResource(defaultResource),
        trace.WithSyncer(exporter),
    ))
}

func installOpenTelemetryExporter() {
    var err error
    defaultResource, err = resource.New(context.Background(),
        resource.WithAttributes(semconv.ServiceNameKey.String("ServiceName")),
        resource.WithFromEnv(),
        resource.WithTelemetrySDK(),
        resource.WithHost(),
    )

    if err != nil {
        fmt.Println(err)
    } else {
        defaultResource = resource.Default()
    }

    installJaegerAsTracerExporter()
    installPrometheusAsMetricExporter()
}

Here is a complete example.

module myapp

require (
    github.com/xgfone/go-apiserver v0.18.0
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.33.0
    go.opentelemetry.io/otel v1.8.0
    go.opentelemetry.io/otel/exporters/jaeger v1.8.0
    go.opentelemetry.io/otel/exporters/prometheus v0.31.0
    go.opentelemetry.io/otel/metric v0.31.0
    go.opentelemetry.io/otel/sdk v1.8.0
    go.opentelemetry.io/otel/sdk/metric v0.31.0
)

go 1.16

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"

    // go-apiserver
    "github.com/xgfone/go-apiserver/entrypoint"
    "github.com/xgfone/go-apiserver/http/reqresp"
    "github.com/xgfone/go-apiserver/http/router"
    "github.com/xgfone/go-apiserver/http/router/routes/ruler"
    "github.com/xgfone/go-apiserver/middleware"

    // OpenTelemetry
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/metric/global"
    controller "go.opentelemetry.io/otel/sdk/metric/controller/basic"
    "go.opentelemetry.io/otel/sdk/metric/export/aggregation"
    processor "go.opentelemetry.io/otel/sdk/metric/processor/basic"
    selector "go.opentelemetry.io/otel/sdk/metric/selector/simple"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

// ------------------------------------------------------------------------ //
// Install the OpenTelemetry exporter. If not, no metric or tracer data is exported.

var defaultResource *resource.Resource

func installPrometheusAsMetricExporter() {
    factory := processor.NewFactory(
        selector.NewWithHistogramDistribution(),
        aggregation.CumulativeTemporalitySelector(),
    )

    ctrl := controller.New(factory, controller.WithResource(defaultResource))
    exporter, err := prometheus.New(prometheus.Config{}, ctrl)
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    ruler.DefaultRouter.Path("/metrics").GET(exporter)
    global.SetMeterProvider(exporter.MeterProvider())
}

func installJaegerAsTracerExporter() {
    exporter, err := jaeger.New(jaeger.WithAgentEndpoint())
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    otel.SetTracerProvider(trace.NewTracerProvider(
        trace.WithResource(defaultResource),
        trace.WithSyncer(exporter),
    ))
}

func installOpenTelemetryExporter() {
    var err error
    defaultResource, err = resource.New(context.Background(),
        resource.WithAttributes(semconv.ServiceNameKey.String("ServiceName")),
        resource.WithFromEnv(),
        resource.WithTelemetrySDK(),
        resource.WithHost(),
    )

    if err != nil {
        fmt.Println(err)
    } else {
        defaultResource = resource.Default()
    }

    installJaegerAsTracerExporter()
    installPrometheusAsMetricExporter()
}

// ------------------------------------------------------------------------ //
// Initialize the HTTP client and server middleware to support OpenTelemetry.

func initHTTPClient(opts ...otelhttp.Option) {
    http.DefaultClient = &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport, opts...),
    }
}

func initHTTPMiddleware() {
    router.DefaultRouter.Middlewares.Use(
        // Add the middleware to let OpenTelemetry wrap the request to handle metric and tracer.
        middleware.NewMiddleware("otel", 100, func(h interface{}) interface{} {
            return otelhttp.NewHandler(h.(http.Handler), "HTTPServerName")
        }),

        // TODO: Add other middlewares...
    )
}

func init() {
    installOpenTelemetryExporter()

    initHTTPClient()
    initHTTPMiddleware()
}

func main() {
    // Add The routes into the router.
    ruler.DefaultRouter.Path("/path1").GET(http.DefaultServeMux)
    ruler.DefaultRouter.Path("/path2").GETFunc(func(w http.ResponseWriter, r *http.Request) {
        // TODO
    })
    ruler.DefaultRouter.Path("/path3").GETContext(func(c *reqresp.Context) {
        // TODO
    })

    startHTTPServer("127.0.0.1:80")
}

func startHTTPServer(addr string) {
    ep, err := entrypoint.NewEntryPoint("", addr, router.DefaultRouter)
    if err != nil {
        log.Fatalf("fail to start the http server")
    }

    go waitSignal(ep.Stop)
    ep.Start()
}

func waitSignal(stop func()) {
    ch := make(chan os.Signal, 1)
    signal.Notify(ch,
        os.Interrupt,
        syscall.SIGTERM,
        syscall.SIGQUIT,
        syscall.SIGABRT,
        syscall.SIGINT,
    )
    <-ch
    stop()
}

[xgfone]

We have no need to write too codes but installing or initializing the metric and tracer exporters.

[xgfone]

I packages the codes above into the repository github.com/xgfone/go-opentelemetry. See Example.

[arpitjindal97]

Awesome man, I will give it a try and will let you know