httpstats client + httpcache + go-github

[roscopecoltran]

Hi guys,

Hope you are all well !

Global overview: I forked a small utility for managing my starred repositories on github.com called https://github.com/hoop33/limo.

I recently added some new go-github api calls, httpcache to make some conditional requests and optimize the use of the api rate limits. Also, i create my own backend to save all responses from the api and compress then with snappy into a badger KB datastore.

Why related to stats/httpstats ? I just wanted to benchmark and grab statistics of my http requests, and get a benchmark of the httpcache or the latency of some paginated http requests. And add some golbal statistics like described in this project https://github.com/jamiealquiza/tachymeter

What is the issues ? I tried to hijack the http.client transport for httpcache or github-go package but I get this error due to the httpstats transport wrapper.

    hcache, err = httpcache_badger.New(
        &httpcache_badger.Config{
            ValueDir:    "api.github.com.v3.gzip",
            StoragePath: cacheStoragePrefixPath,
            SyncWrites:  true,
            Debug:       false,
            Compress:    true,
        })

    // httpcache transport
    t := httpcache.NewTransport(hcache)
    t.MarkCachedResponses = true
    t.Debug = false
    // t.Transport = httpstats.NewTransport(http.DefaultClient.Transport)
    t.Transport = httpstats.NewTransport(http.DefaultTransport)

    // http client
    timeout := time.Duration(10 * time.Second)
    hc = &http.Client{
        Transport: &oauth2.Transport{
            Base:   t,
            Source: ts,
        },
        Timeout: timeout,
    }

    // github client
    ghClient := github.NewClient(hc)

{
"start": "2017-12-26 01:05:05.886", 
"eng": {"Handler":{},"Prefix":"main","Tags":null}, 
"resError": "json: unsupported type: func() (io.ReadCloser, error)", 
"metrics": {}, 
"body": {}, 
"op": "read"
}

httpstats triggers those alarms on Content-Types like "application/json" or "application/octet-stream"

Questions ?

• Is there a way to manage both json.Encoder and io.ReadCloser without being too intrusive into the stats project ?
• Is there a way to expose directly httpstats to an internal process without using a third-party client ? More precisely, I would like to forward metrics to TUI based dashboard. (eg. https://github.com/andreaskoch/gargantua/blob/develop/dashboard.go)

Thanks in advance for any hints or help about this issue ^^.

Cheers, Richard

[achille-roussel]

Hello, thanks for reporting the issue.

First of all, can you confirm that your code works without the httpstats wrapper? For example, if you comment out t.Transport = httpstats.NewTransport(t.Transport), does everything work as expected.

httpstats does not do any serialization of the request body to JSON. The error you get on func() (io.ReadCloser, error) probably comes from a different layer. The signature matches the signature of the GetBody field on http.Request

        // GetBody defines an optional func to return a new copy of
        // Body. It is used for client requests when a redirect requires
        // reading the body more than once. Use of GetBody still
        // requires setting Body.
        //
        // For server requests it is unused.
        GetBody func() (io.ReadCloser, error)

Is it possible that something is attempting to serialize the entire HTTP request object in JSON and fails when it reaches this field?

I would need to have access to a bit more code to understand the problem, at best a runnable program that highlights the issue so I can dig more.

As to producing to a TUI dashboard, there's no such thing in httpstats yet. There are multiple destinations supported to produce metrics to (like datdog, influxdb, etc...), so if you wanted to add a new one I'd be happy to review and accept a pull request.

[roscopecoltran]

Hi,

Thanks for the reply ! :-)

Repos:

• https://github.com/sniperkit/limo (limo + httpcache + stats)
• https://github.com/sniperkit/httpcache (httpcache + badger)

Implementations:

• https://github.com/sniperkit/limo/blob/httpstats/service/service.go#L27-L50 (stats + influxdb)
• https://github.com/sniperkit/limo/blob/httpstats/service/github.go#L186-L221 (Transport Hijack)

Run:

1. Import Deps (Note: other deps are already available in the vendor dir)

   cd $GOPATH/src/github.com/sniperkit/limo
   go get -v github.com/sniperkit/httpcache

2. Register your github personal token (Note: Limit to 5000 requests)

   cd $GOPATH/src/github.com/sniperkit/limo
   go get -v github.com/sniperkit/httpcache
   go run main.go login 

3. Fetch your starred repo (Ref: https://github.com/sniperkit/limo/blob/httpstats/service/github.go#L54-L91)

   go run main.go update

Notes:

• I have added logrus to debug httpstats but maybe you have a better way to output logs from httpstats ^^

Cheers, Richard

[roscopecoltran]

Hi,

I made another examples at:

• https://github.com/sniperkit/cache/blob/sniperkit/examples/cache_stats/main.go

Hope it will be more helpful !

Cheers, Richard

[roscopecoltran]

Hi,

You can close the issue. I have fixed the problem as it was on my side.

Ps. Have you been in Epitech Lyon ?

Cheers, Richard

[achille-roussel]

Good to know.

I was at Epitech in Nice actually.