olivere
commented
4 years ago The error message indicates that there's an issue with signing the request, something related to Amazon services only:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
FATA[0003] error in downloading alerts: elastic: Error 403 (Forbidden) index_name=".opendistro-alerting-alerts*"As already explained in #1317, the commit in https://github.com/olivere/elastic/commit/5c0861c21dfa010ad37a270bd6cc16127deb824c added tests for this behaviour, and it works fine.
Escaping the path is the correct way to do things, otherwise you'd have issues with documents including e.g. a / which, if not escaped, might be interpreted by ES as a call to a different endpoint.
The curl tool will not do any escaping at all btw.
Let's try to fix this. But my gut feeling is that escaping isn't actually the thing that breaks this, but the signing part. I currently do not have access to an Amazon ES cluster, so that may take a while to debug.
ishanjain28
Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.
Which version of Elastic are you using?
[ x ] elastic.v7 (for Elasticsearch 7.x) [ ] elastic.v6 (for Elasticsearch 6.x) [ ] elastic.v5 (for Elasticsearch 5.x) [ ] elastic.v3 (for Elasticsearch 2.x) [ ] elastic.v2 (for Elasticsearch 1.x)
(Didn't check this problem with other versions)
Please describe the expected behavior
It should've returned a response corresponding to this JSON,
Please describe the actual behavior
403 access denied errors.
``` [[ELASTIC]]POST /.opendistro-alerting-alerts%2A/_search?ignore_unavailable=true&scroll=5m&size=500 HTTP/1.1 Host: es-host.com User-Agent: elastic/7.0.17 (linux-amd64) Transfer-Encoding: chunked Accept: application/json Content-Type: application/json Accept-Encoding: gzip 20 {"_source":true,"sort":["_doc"]} 0 [[ELASTIC]]HTTP/2.0 403 Forbidden Content-Length: 192 Access-Control-Allow-Origin: * Content-Type: application/json Date: Sat, 20 Jun 2020 12:03:57 GMT X-Amzn-Requestid: f07eec9f-4291-43e8-9648-41987d0756d9 {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."} FATA[0003] error in downloading alerts: elastic: Error 403 (Forbidden) index_name=".opendistro-alerting-alerts*" ```Any steps to reproduce the behavior?
Yes! Here is a program to replicate this.
```go package main import ( "context" "fmt" stdlog "log" "os" eaws "github.com/olivere/elastic/aws" "github.com/olivere/elastic/v7" log "github.com/sirupsen/logrus" awsauth "github.com/smartystreets/go-aws-auth" ) const ( // IndexName points to the index that is monitored by this function IndexName string = ".opendistro-alerting-alerts*" ChunkSize int = 500 srcESAddr string = "" // ES Cluster Addr // Method of authentication doesn't effect the response. // ie. you can also replicate it if you only have a whitelisted IP and don't want to authorize via the elastic client accessKey string = "" // AWS Access Key secretKey string = "" // AWS Secret key ) func main() { signingClient := eaws.NewV4SigningClient(awsauth.Credentials{ AccessKeyID: accessKey, SecretAccessKey: secretKey, }) srclient, err := elastic.NewClient( elastic.SetURL(srcESAddr), elastic.SetSniff(false), elastic.SetHealthcheck(false), elastic.SetHttpClient(signingClient), elastic.SetErrorLog(stdlog.New(os.Stderr, "ELASTIC ", stdlog.LstdFlags)), elastic.SetInfoLog(stdlog.New(os.Stdout, "", stdlog.LstdFlags)), elastic.SetTraceLog(stdlog.New(os.Stderr, "[[ELASTIC]]", 0)), ) if err != nil { log.Fatalln(err) } req := elastic. NewScrollService(srclient). Index(IndexName). FetchSource(true). Size(ChunkSize). SortBy(elastic.SortByDoc{}). IgnoreUnavailable(true) resp, err := req.Do(context.Background()) if err != nil { log.WithFields(log.Fields{ "index_name": IndexName, }).Fatalf("error in downloading alerts: %v", err) } fmt.Println(resp) } ```The issue here is the fact that the library escapes
*into%2A. If I manually send a curl request to/.opendistro-alerting-alerts*/_search?ignore_unavailable=true&scroll=5m&size=500I get the expected result.Also, ScrollService is not the only place where we have this problem. Other functions in the code also have this problem. See #1317 for some more examples.