AWS Timestream UnknownOperationException

[khinkson]

Describe the bug I am going to assume I am doing something wrong, but I've exhausted my debugging attempts. When attempting to use SotoTimestreamQuery or SotoTimestreamWrite I get the following error:

Unhandled error, code: notFound, body: {"__type":"com.amazon.coral.service#UnknownOperationException"}

To Reproduce Steps to reproduce the behavior:

1. Create a 'swift-aws-lambda-runtime' project. Setup environment variables
2. Add SotoTimestreamQuery and query a working database

See below for a link and attached project.

Expected behavior For the query to return something like below as a response: { "Rows": [], "ColumnInfo": [ { "Name": "measure_name", "Type": { "ScalarType": "VARCHAR" } }, { "Name": "time", "Type": { "ScalarType": "TIMESTAMP" } } ], "QueryId": "AECACAM4KDBPWS364MYCUTB6R7WM2FF72HRZYQCLPT5EV4BJORMN3BNQZAYTYGA", "QueryStatus": { "ProgressPercentage": 100.0, "CumulativeBytesScanned": 0, "CumulativeBytesMetered": 10000000 } }

Setup (please complete the following information):

• OS: macOS 11.3.1, Xcode 12.5, Swift 5.4
• Version of soto: 5.5.0 with soto-core 5.3.0
• Authentication mechanism: credentials in ~/.aws/credentials and auth using AWS_PROFILE

Additional context Region used: us-east-2

I can query the database via the web console and also from the command line using the aws-cli. The setup project is also similarly setup to other I use to access DynamoDB, SQS & S3

Link & attached project TimestreamTest-main.zip Or link: TimestreamTest Github Project

[adam-fowler]

Hi,

I had a look at this and for TimeStream you need to use a custom endpoint. The only operation that works with the standard endpoint is DescribeEndpoints. This then gives you an endpoint and the amount of time it is valid. See https://docs.aws.amazon.com/timestream/latest/developerguide/Using-API.endpoint-discovery.how-it-works.html

You create a new TimestreamQuery service object with this endpoint

let tsQuery = TimestreamQuery(client: awsClient, endpoint: "https://\(endpoint)")

You can run queries on the tsQuery service object.

Unfortunately there is a bug in Soto such that the endpoint discovery request is invalid. I will be fixing this but in the meantime you can get around it by adding a custom middleware for your timestream service object that does the endpoint discovery.

struct TSEndpointMiddleware: AWSServiceMiddleware {
    func chain(request: AWSRequest, context: AWSMiddlewareContext) throws -> AWSRequest {
        var request = request
        request.body = .text("{}")
        return request
    }
}
let tsEndpointDiscovery = TimestreamQuery(
    client: awsClient, 
    region: <your-region>
).with(middlewares: [TSEndpointMiddleware()])

Do not use this on the second Timestream service object as it will break queries by overwriting them.

[khinkson]

Thank you.

Going over the above, I was able to make the query work.

I had been reading the section of the documentation where they recommend against using the describe endpoint call manually (with exceptions that now make more sense given the above)

To me it looks like endpoints should be managed transparently without being need to be passed in. Is that a planned feature or is it already in there and I missed it? Do the official AWS SDKs work by transparently fetching and caching the endpoint?

[adam-fowler]

As the majority of the Soto service libraries are auto generated from the json model files AWS publishes there is generally a one to one mapping from the AWS REST apis to the apis Soto provides. So out of the box it doesn't create more complex patterns of service discovery, like Timestream requires.

But we do have an extensions folder where we have added code specific for certain services. I could add endpoint management code in there. It doesn't look like it would be overly complex. Let me investigate.

[khinkson]

Awesome. I was thinking about how to contribute to it myself once the describe bug was resolved.

I would have to think through the timeout cache management for an endpoint. I'm not exactly sure how the API behaves there on the server side.

I'll keep an eye out for your changes. Thanks again.

[adam-fowler]

@khinkson SotoCore v5.3.1 fixes the Timestream.DescribeEndpoints issue

[khinkson]

@adam-fowler Thank you. I'll check it out.