[BUG] "The request signature we calculated does not match the signature you provided." error on AOS benchmark query step

[junqiu-lei]

Describe the bug I am using the OSB 1.1.0 to run Amazon OpenSearch benchmarks, it can create, ingest index, but got error at the query step.

I found the search api call in OSB is using GET method, but in the documentation https://docs.aws.amazon.com/opensearch-service/latest/developerguide/searching.html#searching-dsl, it says:

The _search API accepts HTTP GET and POST for request body searches, 
but not all HTTP clients support adding a request body to a GET request. POST is the more universal choice.

Similar issue: https://stackoverflow.com/questions/69869225/why-do-i-get-the-request-signature-we-calculated-does-not-match-the-signature-y/69913289#69913289

So I am wondering if we need update search call with POST method?

To Reproduce

    opensearch-benchmark \
        execute-test \
        --distribution-version 2.9.0 \
        --target-hosts $URL:$PORT \
        --pipeline benchmark-only \
        --workload $workload_path \
        --workload-params $params_path \
        --client-options timeout:$client_timeout,amazon_aws_log_in:client_option,aws_access_key_id:$AWS_ACCESS_KEY_ID,aws_secret_access_key:$AWS_SECRET_ACCESS_KEY,region:$AWS_DEFAULT_REGION,service:$OSB_SERVICE \
        --test-procedure no-train-test \
        --results-file $osb_local_fixed_output_file \
        --results-format csv \
        --results-file $osb_result_file

Expected behavior No query error

Logs

2023-10-16 19:33:40,158 -not-actor-/PID:26414 opensearch WARNING GET [https://my-cluster:443/target_index/_search?_source=%7B%27exclude%27%3A+%5B%27target_field%27%5D%7D](https://my-cluste/target_index/_search?_source=%7B%27exclude%27%3A+%5B%27target_field%27%5D%7D) [status:403 request:0.037s]
2023-10-16 19:33:40,159 -not-actor-/PID:26414 opensearch DEBUG > {"size":100,"query":{"knn":{"target_field":{"vector":[1.0,3.0,11.0,110.0,62.0,22.0,4.0,0.0,43.0,21.0,22.0,18.0,6.0,28.0,64.0,9.0,11.0,1.0,0.0,0.0,1.0,40.0,101.0,21.0,20.0,2.0,4.0,2.0,2.0,9.0,18.0,35.0,1.0,1.0,7.0,25.0,108.0,116.0,63.0,2.0,0.0,0.0,11.0,74.0,40.0,101.0,116.0,3.0,33.0,1.0,1.0,11.0,14.0,18.0,116.0,116.0,68.0,12.0,5.0,4.0,2.0,2.0,9.0,102.0,17.0,3.0,10.0,18.0,8.0,15.0,67.0,63.0,15.0,0.0,14.0,116.0,80.0,0.0,2.0,22.0,96.0,37.0,28.0,88.0,43.0,1.0,4.0,18.0,116.0,51.0,5.0,11.0,32.0,14.0,8.0,23.0,44.0,17.0,12.0,9.0,0.0,0.0,19.0,37.0,85.0,18.0,16.0,104.0,22.0,6.0,2.0,26.0,12.0,58.0,67.0,82.0,25.0,12.0,2.0,2.0,25.0,18.0,8.0,2.0,19.0,42.0,48.0,11.0],"k":100}}}}
2023-10-16 19:33:40,159 -not-actor-/PID:26414 opensearch DEBUG < b'{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\\n\\nThe Canonical String for this request should have been\\n\'GET\\n/target_index/_search\\n_source=%7B%27exclude%27%3A%20%5B%27target_field%27%5D%7D\\nhost:search-parallel-cluster-test-1-at6awh3ezbtu6ulpzfhe24ajxq.us-east-1.es.amazonaws.com\\nx-amz-date:20231016T193340Z\\n\\nhost;x-amz-date\\n7fb889f588571c30fb35ec33f6f8dd197646ddce965897a634974f5f2966ac83\'\\n\\nThe String-to-Sign should have been\\n\'AWS4-HMAC-SHA256\\n20231016T193340Z\\n20231016/us-east-1/es/aws4_request\\n296da7b40dee9447d15005b57776c2f80ca412c3d08f1cff9fe1ab7dba193a8f\'\\n"}'

More Context (please complete the following information):

• Workload
  • https://github.com/opensearch-project/k-NN/blob/main/benchmarks/osb/workload.json
  • https://github.com/opensearch-project/k-NN/blob/main/benchmarks/osb/procedures/no-train-test.json
• Service: AOS
• Version 2.9

Additional context Add any other context about the problem here.

[navneet1v]

@IanHoang I tested the opensearch-py client with a simple python script and found out that this is an issue with OSB only, as the below script ran successfully

from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3

host = '<DOMAIN-NAME>' # cluster endpoint, for example: my-test-domain.us-east-1.es.amazonaws.com
region = 'us-east-1'
service = 'es' # 'aoss' for OpenSearch Serverless
credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region, service)

client = OpenSearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = auth,
    use_ssl = True,
    verify_certs = True,
    connection_class = RequestsHttpConnection,
    pool_maxsize = 20
)

print(client.info())

index_name = 'test-index'
print("Creating index")
print(client.indices.create(index_name))

q = 'miller'

query = {
    'size': 5,
    'query': {
        'multi_match': {
            'query': q,
            'fields': ['title^2', 'director']
        }
    }
}

response = client.transport.perform_request("GET", f"/{index_name}/_search", body=query)

# response = client.search(
#     body = query,
#     index = index_name
# )

print('\nSearch results:')
print(response)

print("Deleting Index")
print(client.indices.delete(index_name))