soto-project / soto

Swift SDK for AWS that works on Linux, macOS and iOS
https://soto.codes
Apache License 2.0
880 stars 83 forks source link

AWS Timestream UnknownOperationException #494

Closed khinkson closed 3 years ago

khinkson commented 3 years ago

Describe the bug I am going to assume I am doing something wrong, but I've exhausted my debugging attempts. When attempting to use SotoTimestreamQuery or SotoTimestreamWrite I get the following error:

Unhandled error, code: notFound, body: {"__type":"com.amazon.coral.service#UnknownOperationException"}

To Reproduce Steps to reproduce the behavior:

  1. Create a 'swift-aws-lambda-runtime' project. Setup environment variables
  2. Add SotoTimestreamQuery and query a working database

See below for a link and attached project.

Expected behavior For the query to return something like below as a response: { "Rows": [], "ColumnInfo": [ { "Name": "measure_name", "Type": { "ScalarType": "VARCHAR" } }, { "Name": "time", "Type": { "ScalarType": "TIMESTAMP" } } ], "QueryId": "AECACAM4KDBPWS364MYCUTB6R7WM2FF72HRZYQCLPT5EV4BJORMN3BNQZAYTYGA", "QueryStatus": { "ProgressPercentage": 100.0, "CumulativeBytesScanned": 0, "CumulativeBytesMetered": 10000000 } }

Setup (please complete the following information):

Additional context Region used: us-east-2

I can query the database via the web console and also from the command line using the aws-cli. The setup project is also similarly setup to other I use to access DynamoDB, SQS & S3

Link & attached project TimestreamTest-main.zip Or link: TimestreamTest Github Project

adam-fowler commented 3 years ago

Hi,

I had a look at this and for TimeStream you need to use a custom endpoint. The only operation that works with the standard endpoint is DescribeEndpoints. This then gives you an endpoint and the amount of time it is valid. See https://docs.aws.amazon.com/timestream/latest/developerguide/Using-API.endpoint-discovery.how-it-works.html

You create a new TimestreamQuery service object with this endpoint

let tsQuery = TimestreamQuery(client: awsClient, endpoint: "https://\(endpoint)")

You can run queries on the tsQuery service object.

Unfortunately there is a bug in Soto such that the endpoint discovery request is invalid. I will be fixing this but in the meantime you can get around it by adding a custom middleware for your timestream service object that does the endpoint discovery.

struct TSEndpointMiddleware: AWSServiceMiddleware {
    func chain(request: AWSRequest, context: AWSMiddlewareContext) throws -> AWSRequest {
        var request = request
        request.body = .text("{}")
        return request
    }
}
let tsEndpointDiscovery = TimestreamQuery(
    client: awsClient, 
    region: <your-region>
).with(middlewares: [TSEndpointMiddleware()])

Do not use this on the second Timestream service object as it will break queries by overwriting them.

khinkson commented 3 years ago

Thank you.

Going over the above, I was able to make the query work.

I had been reading the section of the documentation where they recommend against using the describe endpoint call manually (with exceptions that now make more sense given the above)

To me it looks like endpoints should be managed transparently without being need to be passed in. Is that a planned feature or is it already in there and I missed it? Do the official AWS SDKs work by transparently fetching and caching the endpoint?

adam-fowler commented 3 years ago

As the majority of the Soto service libraries are auto generated from the json model files AWS publishes there is generally a one to one mapping from the AWS REST apis to the apis Soto provides. So out of the box it doesn't create more complex patterns of service discovery, like Timestream requires.

But we do have an extensions folder where we have added code specific for certain services. I could add endpoint management code in there. It doesn't look like it would be overly complex. Let me investigate.

khinkson commented 3 years ago

Awesome. I was thinking about how to contribute to it myself once the describe bug was resolved.

I would have to think through the timeout cache management for an endpoint. I'm not exactly sure how the API behaves there on the server side.

I'll keep an eye out for your changes. Thanks again.

adam-fowler commented 3 years ago

@khinkson SotoCore v5.3.1 fixes the Timestream.DescribeEndpoints issue

khinkson commented 3 years ago

@adam-fowler Thank you. I'll check it out.