prometheus / client_golang

Prometheus instrumentation library for Go applications
https://pkg.go.dev/github.com/prometheus/client_golang
Apache License 2.0
5.34k stars 1.17k forks source link

Support for disabling partial responses via API client #880

Open metalmatze opened 3 years ago

metalmatze commented 3 years ago

Hey, I'm currently using these two functions to query some data from Thanos with this API client.

https://github.com/prometheus/client_golang/blob/196536534fbba6249b94d4ccb2269cf162ceef44/api/prometheus/v1/api.go#L803 https://github.com/prometheus/client_golang/blob/196536534fbba6249b94d4ccb2269cf162ceef44/api/prometheus/v1/api.go#L821

Now, I don't want partial responses from Thanos but rather want to fail the requests. Therefore I need to add a ?partial_response=false query parameter to the request and I don't think that's currently possible with that API client.

I'm wondering if we should have it in client_golang? :thinking_face: It not hard to do these requests "manually" either.

metalmatze commented 3 years ago

Looking a bit more into this I have for now wrapped the small Client interface with a middleware.

// thanosClient wraps the Prometheus Client to inject some headers to disable partial responses
// and enables querying for downsampled data.
type thanosClient struct {
    client api.Client
}

func (c *thanosClient) URL(ep string, args map[string]string) *url.URL {
    return c.client.URL(ep, args)
}

func (c *thanosClient) Do(ctx context.Context, r *http.Request) (*http.Response, []byte, error) {
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        return nil, nil, err
    }
    query, err := url.ParseQuery(string(body))
    if err != nil {
        return nil, nil, err
    }

    // We don't want partial responses, especially not when calculating error budgets.
    query.Set("partial_response", "false")
    r.ContentLength += 23

    // I add a few more things I excluded from this example.

    encoded := query.Encode()
    r.Body = ioutil.NopCloser(strings.NewReader(encoded))
    return c.client.Do(ctx, r)
}

This should get me pretty far. Still I'm wondering if we couldn't allow to pass query params from the outside so I wouldn't need to encode and decode the body another time.

yeya24 commented 1 year ago

I am interested in this one, too. Generally, it would be great for the client to provide extensibilities to inject other parameters/headers to the path for use cases like Thanos and Cortex.

bofeng96 commented 1 year ago

I am interested in this one, too. Generally, it would be great for the client to provide extensibilities to inject other parameters/headers to the path for use cases like Thanos and Cortex.

Hey Ben. Can I assume that all prometheus client libraries are enabling partial response for Thanos behind the scenes?

yeya24 commented 1 year ago

@bofeng96 At least this official Prometheus client doesn't support this. I am not sure about other client libraries but this is the official one. Seems the only way now is to add some custom http transport or write your own library to set the param.

Generally Thanos Querier has 2 ways to configure partial response.

  1. Set partial_response=true per query
  2. Set --query.partial-response=true flag when running Thanos Querier.