Support Amazon managed prometheus with AWS Sigv4

[gavin-jeong]

I wanted use Trickster for Amazon managed prometheus. And it requires AWS SigV4 Auth https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-query-APIs.html.

For this I wrote some PR here https://github.com/trickstercache/trickster/pull/683

Seems it works with curl. But when we take the Trickster as Grafana Datasource, there are some problem.

1. Failed register the trickster to Datasource. Grafana tries check /api/v1/status/buildinfo. But with Trickster and AMP, it returns 404.

2. When we query from Grafana, AMP returns Error The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

Error log from Trickster

2023/07/30 04:27:40 /api/v1/query_range?end=1690658850&query=1%2B1&start=1690655250&step=15
time=2023-07-29T19:27:41.384532Z app=trickster caller=proxy/engines/deltaproxycache.go:481 level=error event="unexpected upstream response" clientRequestMethod=GET upstreamRequestURL="https://aps-workspaces.ap-northeast-1.amazonaws.com/workspaces/ws-224d07f9-9fc3-4b6f-9ab9-f1b5be37fbc1/api/v1/query_range?end=1690658850&query=1%2B1&start=1690655250&step=15" upstreamRequestMethod=GET upstreamRequestHeaders="{[Authorization:AWS4-HMAC-SHA256 Credential=ASIAUSRUJXJVJ7KD3B67/20230729/ap-northeast-1/aps/aws4_request, SignedHeaders=forwarded;host;via;x-amz-date;x-amz-security-token;x-datasource-uid;x-grafana-org-id;x-panel-id, Signature=cba8e3bcd3744d2b7904e9e31a9a6e5ff8636ae97d2b8de4514bd68fc10bd00a],[Forwarded:for=127.0.0.1;proto=https],[User-Agent:Grafana/10.0.1],[Via:HTTP/1.1 Gavin.Jeong's-MacBook-Pro],[X-Amz-Date:20230729T192740Z],[X-Amz-Security-Token:FwoGZXIvYXdzEA0aDEBMlXwwWi+v3SRaeCK7AXEuk0GiU7VBRQgbGFBkpOZP7XOFiGs5XlocIGvRH5uKxvBPNoobT73OZCINOrLrMVLyweAuvQgJ71IoHevjwWMD1UNLypCuQekP1bNXsGNAO/Sa/wjcGROHX+LgY8zSZUUWR50xvTf6O04kRkxIuFcfC3XYGSfjgVJmWyBLuvdwL+Yxy5iwf3IOrixWbKSB5ALvcKtuaHkcXnQJyp9rzP6B6QPhomGp8YquD8zO68Cl1dauSK2ro8MNjMkordCVpgYyLRf1fMu5GltI2QRixhFOkcp04P4r8b+Z+ML81g4/7yWunonTw7Kn8KqICWKZCg==],[X-Datasource-Uid:ce7e4996-01b6-4f43-b351-b28e020d56a2],[X-Grafana-Org-Id:1],[X-Panel-Id:Q-89d59cd1-8bf5-47a4-8ad6-c46e1166d03a-0]}" upstreamResponseHeaders="{[Cache-Control:s-maxage=21600],[Connection:keep-alive],[Content-Type:application/json],[Date:Sat, 29 Jul 2023 19:27:41 GMT],[Server:amazon],[X-Amzn-Errortype:InvalidSignatureException],[X-Amzn-Requestid:72bdd91d-77a2-442d-a148-a85b54b43893]}" upstreamResponseBody="{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\\n\\nThe Canonical String for this request should have been\\n'GET\\n/workspaces/ws-****/api/v1/query_range\\nend=1690658850&query=1%2B1&start=1690655250&step=15\\nforwarded:for=127.0.0.1;proto=https\\nhost:aps-workspaces.ap-northeast-1.amazonaws.com\\nvia:HTTP/1.1 ****\\nx-amz-date:20230729T192740Z\\nx-amz-security-token:****\\nx-datasource-uid:\\nx-grafana-org-id:1\\nx-panel-id:Q-89d59cd1-8bf5-47a4-8ad6-c46e1166d03a-0\\n\\nforwarded;host;via;x-amz-date;x-amz-security-token;x-datasource-uid;x-grafana-org-id;x-panel-id\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\\n\\nThe String-to-Sign should have been\\n'AWS4-HMAC-SHA256\\n20230729T192740Z\\n20230729/ap-northeast-1/aps/aws4_request\\****'\\n\"}" clientRequestURL="https://aps-workspaces.ap-northeast-1.amazonaws.com/workspaces/ws-****/api/v1/query_range?end=1690658850&query=1%2B1&start=1690655250&step=15" clientRequestHeaders="unsupported value type" statusCode=403

Seems Grafana generating signature based on the HTTP header, and AMP check this. May be it is for preventing MITM like attacks.

• https://github.com/grafana/grafana/blob/main/pkg/tsdb/prometheus/client/transport.go#L36
• https://github.com/prometheus/common/blame/main/sigv4/sigv4.go#L128

I don't know well with the Trickster, what I tries are just return some json data from /api/v1/status/buildinfo and removes all the client headers from DoProxy(). Then it looks just work. But not sure it is right approach. I had tried using paths and request_rewrites also but not worked as expected.

Anyone can share some idea?

[jranson]

@gavin-jeong thanks so much for contributing this feature!