Unable to connect to Timestream from using endpoint (.Net)

Anna-Ignatov commented 3 months ago

Describe the bug

Trying to connect to Timestream from .Net using query endpoint - we are using private VPC

Expected Behavior

Connect to Timestream

Current Behavior

This is configuration we sent: RegionEndpoint = RegionEndpoint.EUWest1, Timeout = tsConfig.Timeout, MaxErrorRetry = tsConfig.MaxErrorRetry, EndpointDiscoveryEnabled=true
And received Amazon.Runtime.AmazonClientException: Failed to discover the endpoint for the request. Requests will not succeed until an endpoint can be retrieved or an endpoint is manually specified We tried to specify also ServiceURL= "https://query-cell1.timestream.eu-west-1.amazonaws.com" And received the same error. We also tried to set EndpointDiscoveryEnabled= false - with the same result In case we do no specify service URL we receive The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing

Reproduction Steps

var awsConfig=new AmazonTimestreamQueryConfig { RegionEndpoint = RegionEndpoint.EUWest1, Timeout = tsConfig.Timeout, MaxErrorRetry = tsConfig.MaxErrorRetry, EndpointDiscoveryEnabled=false // ServiceURL= "https://query-cell1.timestream.eu-west-1.amazonaws.com"
}; Services.AddScopedAsImplementedInterfaces(awsConfig);

We have a log of configuration: public AwsTimeSeriesRepository(TimeSeriesConfig config, IAmazonTimestreamQuery timestreamClient, ILogger logger) { _timestreamQueryBuilder = new TimestreamQueryBuilder(config); _timestreamClient = timestreamClient; _logger = logger; _config = config; _logger.LogInformation("config={config}", JsonConvert.SerializeObject(_config)); _logger.LogInformation("AWSConfig={awsConfig}", JsonConvert.SerializeObject(timestreamClient.Config)); }

        Please, see the AWS configuration we received in log:

"ServiceId": "Timestream Query", "RegionEndpointServiceName": "query.timestream", "ServiceVersion": "2018-11-01", "UserAgent": "aws-sdk-dotnet-coreclr/3.7.301.22 ua/2.0 os/linux#5.10.217.205 md/ARCH#X64 lang/.NET_Core#7.0.20 md/aws-sdk-dotnet-core#3.7.304.17 api/Timestream Query#3.7.301.22", "Profile": null, "AWSTokenProvider": { "ProfileName": null }, "SignatureMethod": 1, "SignatureVersion": "4", "UseAlternateUserAgentHeader": false, "RegionEndpoint": null, "ServiceURL": " https://query-cell1.timestream.eu-west-1.amazonaws.com/" , "UseHttp": false, "AuthenticationRegion": null, "AuthenticationServiceName": "timestream", "MaxErrorRetry": 1, "IsMaxErrorRetrySet": true, "LogResponse": false, "ReadEntireResponse": false, "BufferSize": 8192, "ProgressUpdateInterval": 102400, "ResignRetries": false, "AllowAutoRedirect": true, "LogMetrics": false, "DisableLogging": false, "DefaultConfigurationMode": 5, "ProxyCredentials": null, "Timeout": "1.00:00:00", "UseDualstackEndpoint": false, "UseFIPSEndpoint": false, "IgnoreConfiguredEndpointUrls": false, "DisableRequestCompression": false, "RequestMinCompressionSizeBytes": 10240, "ClientAppId": null, "ThrottleRetries": true, "CorrectedUtcNow": "2024-06-20T12:44:10.776782+00:00", "ClockOffset": "00:00:00", "DisableHostPrefixInjection": false, "EndpointDiscoveryEnabled": false, "EndpointDiscoveryCacheLimit": 1000, "RetryMode": 0, "FastFailRequests": false, "CacheHttpClient": true, "HttpClientCacheSize": 1, "ReadWriteTimeout": null, "EndpointProvider": {}, "ProxyHost": null, "ProxyPort": 0, "MaxConnectionsPerServer": null, "HttpClientFactory": null }

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.TimestreamQuery" Version="3.7.301.22"

Targeted .NET Platform

.Net7

Operating System and version

AmazonLinux

ashishdhingra commented 3 months ago

@Anna-Ignatov Good morning. Thanks for opening the issue. Service endpoints lists the service endpoints for TimeStream. Could you try or confirm below:

Enable verbose logging using code below and share/examine the detailed log for exact error:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

You may develop a simple console application and run it on your EC2 instance in private VPC to isolate the configuration.

Check your VPC for security group rules that might be blocking outgoing traffic.
If you have VPC endpoint configured, then refer Configuring VPC endpoint service connectivity (documentation is for KMS service, but should work for Timestream service) to allow service connectivity.

Thanks, Ashish

amraninoam commented 3 months ago

I am getting the same issue as the OP: Trying to connect to timestream with no internet connection over vpc endpoint. I reckon the issue is that the .Net SDK allows configuration of the service url, which I found available only when there's an internet connection. The service url (i.e. https://query.timestream...) is different from endpoint url (i.e. https://query-cell1.timestream... ), which can be accessed internally, without internet connection. Regardless of EndpointDiscoveryEnabled setting true or false, the SDK is calling the service url to find out the endpoint url. Thus, resulting in inability to use the .net sdk in an environment without internet access.

@ashishdhingra, what is the expected way to workaround no internet connectivity?

dscpinheiro commented 3 months ago

@Anna-Ignatov and @amraninoam Did this exception start happening after you updated the AWSSDK.TimestreamQuery package? There was a change on how the SDK resolves endpoints in version 3.7.100, so I'm trying to understand if this was an unintended side effect.

amraninoam commented 3 months ago

@dscpinheiro , I have tried it with the latest SDK, you can find my code here, and the specific SDK versions here I also tried with lower SDKs on a different branch , it result in the same behaviour as 3.7.300.

dscpinheiro commented 3 months ago

Thanks for the repository, it was helpful when we were troubleshooting the problem.

I put out a PR (https://github.com/aws/aws-sdk-net/pull/3367) that was included in today’s release; it has more details but as you mentioned it was related to the custom service URL and an odd TimeStream behavior we weren’t accounting for.

Would you be able to test again with the latest version of the AWSSDK.TimestreamQuery package (which is 3.7.301.29) and let us know if the fix worked for you?

amraninoam commented 3 months ago

Using the repo before and running the new package:

[x] serviceURL=https://query-cell1.timestream.eu-west-1.amazonaws.com works as expected in an environment with no internet. This resolves our issue :tada:
[x] Skipping serviceURL and passing only region works as expected: Calling https://query.timestream.eu-west-1.amazonaws.com first, and then https://query-cell1.timestream.eu-west-1.amazonaws.com
[ ] serviceURL=https://query.timestream.eu-west-1.amazonaws.com fails with exception that IMHO is hard to decipher:

AWS Timestream Query Error: The service returned an error with Error Code UnknownOperationException and HTTP Body: {"__type":"com.amazon.coral.service#UnknownOperationException"}
Error Code: UnknownOperationException
Error Type: Unknown
Request ID: 589eabc6-e0b7-44b1-bddd-38f580a12bed

dscpinheiro commented 3 months ago

Great, glad it solved your issue! I agree that exception is hard to understand, unfortunately TimeStream has this odd behaviour where the operations requiring endpoint discovery (e.g. Query) only work when using the cell endpoints.

I mentioned this in my PR too, but this is one of the reasons we don't use the endpoint discovery for new services (as you noticed, you can run into a few sharp edges).

github-actions[bot] commented 3 months ago

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws / aws-sdk-net